Import Mastodon (Beta)

  • Resolved TheVoidTLMB

    (@thevoidtlmb)


    9 months ago

    I know it’s still beta, but there’s zero documentation (that I can see) for the Import Mastodon feature. (Is this even the right forum to be asking this?) All I can see under Tools>Import>Mastodon(Beta) is “Run importer”, and I don’t want to click that in case it immediately starts importing all the posts from the Mastodon account I’m logged into in my browser.

    I don’t really think that’s what would happen, but with no “Details”, I can’t be sure. What I’d like to know is:

    • Do we get to select a file/archive to use as the import source?
    • If so, which type of file? Hopefully a previously downloaded .zip Mastodon archive.
    • Does every imported Masto post become a single WP post, with any attached media being sent to the WP Media library, and embedded into the post?
    • Does each created WP post have the same publish date as the original Masto post? I hope so. (I don’t want 500 “new” posts at the top of my WP feed)
    • And, while I’d want each post to be “available” via my WP’s APub , I wouldn’t want to push out 500 “new” old posts to my blog’s followers. How would that work?
    • What is the title of each new WP post? Would it be the first sentence or few words of the post, or something dull but sensible like the original post ID number, eg 123456789012345 ? (I can see that the latter might be useful, so you could easily associate, say, myoldinstance.social/@myusername/123456789012345 with
      mywordpresssite.com/blog/123456789012345 )
    • Do Favourite/Boost counts get imported, and/or shown? And who they were by?
    • If it all goes horribly wrong, are there any Undo options?

    Sorry for all the questions! I’m interested in trying it out, but I want to know exactly what I’m trying.

Viewing 15 replies - 16 through 30 (of 36 total)
1 2 3
  • Plugin Author Konstantin Obenland

    (@obenland)

    2 months, 2 weeks ago

    Yes, it looks like an issue with the recently introduced extract_recipients_from_activity_property() function, I’ll take a look!

    I can also look into .tar.gz support (no promises on timeline), I just went with .zip for now as that was what I got back from mastodon.social (or mas.to?).

    The “archive does not contain an Outbox file” is curious. Hard to debug without the zip file. The code that does the processing is pretty basic, so it’s possible it doesn’t catch files that internally are of a different type and show up differently in the filesystem or archives that are in a subdirectory, it doesn’t really “search” for it.

    Thanks so much for trying it out and sharing your feedback!

    Thread Starter TheVoidTLMB

    (@thevoidtlmb)

    2 months, 2 weeks ago

    Ah, great, I’ll wait for un update then. No problem, and thanks to you for the feature in the first place! You have a Mastodon DM re the outbox issue. @obenland

    Thread Starter TheVoidTLMB

    (@thevoidtlmb)

    2 months ago

    Update: I saw that 7.6.1 was out, so I updated AP, and had another go.

    First attempt, using my #1 file from above (the one you have for testing, @obenland ), and after pressing “Import”, I got this error:
    “Sorry, there has been an error.
    Could not insert post into the database.”

    Hmm, not so good. So, second attempt, using my file #4 from above, and I got
    “Imported 4 posts.
    All done. Have fun!”

    Ah, great! But curious. So then I went to look at my list of posts (ordered by PostID), and I see the 6 most recent posts there. But wait, file #4 contains *4* posts (all successfully imported, including one image and one video). The other 2 new posts are from my file #1 – both text-only posts (inc hashtags) – so it looks like the first attempt at an import started off okay but… failed when it got to an image post, maybe? (Checking my media library, the only two new entries there are the ones from file #4)

    [I then successfully edited the Categories of those 2 posts.]

    So… getting there? But I don’t think I want to test with my 2 much larger archives just yet…

    Plugin Author Konstantin Obenland

    (@obenland)

    2 months ago

    Hm, that is frustrating. Thanks for your continued feedback and patience on this.

    I’ll take another look this afternoon

    Plugin Author Konstantin Obenland

    (@obenland)

    1 month, 1 week ago

    I gave this another go around. You won’t be surprised to hear that importing list #1 worked fine in my local environment. 🙂

    “Could not insert post into the database.” points to a database error that WordPress surfaces natively. In a case like this I’d usually recommend trying again. Since the importer skips already imported posts, it should be safe to run again. I suppose an improvement here could be to not just bail with an error but also account for any preceding successful import, to avoid confusion?

    #s1&2 are .tar.gz files, not zip, so on trying #1 I got the error message ~”that’s not a zip file”. 

    I created a draft PR to add support for .tar.gz files. However I don’t feel super comfortable shipping that (at this point) as it requires pretty low level command executions and I’d like to be absolutely sure it’s safe before moving forward.

    All that said, thank you again for sharing your export file with me. There’s nothing more helpful than seeing real world usage and it resulted in two further improvements I was able to make:

    This should make it at least a little bit more useful.

    At this point I’d say, be bold and try one of your bigger archives? The fatal error is fixed and the database error you ran into should only have been temporary. (maybe try it on a test site first?)

    Thread Starter TheVoidTLMB

    (@thevoidtlmb)

    1 month, 1 week ago

    Hi @obenland , thanks for your continued fixes.
    I’ve made slight progress, but…

    Try again to import file #1.
    (First, Export backup file. Done.)

    Straight to “Sorry, there has been an error.
    Could not insert post into the database.”
    No further posts have been imported. Well, this is frustrating, since you’ve had success with the exact same file, and I’ve had success with a different file. Okay, then let’s create a TEST subdomain…

    Went to my host, created a subdomain, and immediately installed WP. The set-up installs WP Super Cache plugin. I installed Sundance theme to match my main set-up. Installed WPApps Show Post ID plugin. Installed ActivityPub, blog profile only. (Yes, there’s the Hello World post in Mastodon).

    Now, in TEST subdomain, I run the importer, choose File (#1), Upload File & Import, Import.
    “Imported 19 posts. All done. Have fun!”
    Okay, this is great. (Those replied-to-posts are showing up embedded nicely, cool!) So the problem on my main site is definitely not a problem with the file, nor the importer.

    Next step – back to my main site, and (just in case…) delete (“Bin”) the two posts that were succesfully imported from file #1. Done. While I’m at it, let’s delete the 4 posts imported from file #4 too. Done. (The two files that the import placed into the media library were not deleted. Hmmm, that’s something to consider/remember. Deleted them manually.)

    Now, let’s try one more time to import file #1 on my main site….
    “Sorry, there has been an error.
    Could not insert post into the database.”
    Again. Looking in Posts, it has not imported any posts at all. sigh

    Be brave? Ok, let’s try file #2 on my TEST subdomain. This one’s quite a bit bigger so there’ll be lots of media files (mostly images) to delete after I’ve finished testing, but sure, let’s go…
    A longer wait (obviously) before the Import button showed up. Click it and…
    “Sorry, there has been an error.
    The archive does not contain an Outbox file, please try again.
    Imported 0 posts.
    All done. Have fun!”
    Dammit. Again, the downloaded archive was a tar.gz , which I expanded, then compressed the resulting folder (which does contain an outbox.json file) to get a zip file, and that zip is the file I tried to import. Tried deleting the zip file from my computer, and compressing the folder once, just in case, to get a new zip file. Tried the importer again in my TEST subdomain:
    “Sorry, there has been an error.
    The archive does not contain an Outbox file, please try again.
    Imported 0 posts.
    All done. Have fun!”
    Well, at least that’s consistent. :-/

    So there are still 2 issues I’m encountering:

    • Does not contain an Outbox error
    • Database error

    Maybe I need to contact my host about the second one, but I’m not sure what conclusion you came to before about the first one?

    Plugin Author Konstantin Obenland

    (@obenland)

    1 month, 1 week ago

    I’m not sure what conclusion you came to before about the [does not contain an Outbox error]?

    None so far. This could be filesystem issue or something in WordPress’ internal filesystem abstraction. The plugin currently grabs a list of all files in extracted folder and looks for outbox.json. I suppose that’s not really needed or even what’s of interest, but rather “can I read that file”. So maybe I should check for that and see if that makes the process more robust. Next on my list of things to try.

    The two files that the import placed into the media library were not deleted. Hmmm, that’s something to consider/remember. Deleted them manually.

    Yes, that is a WordPress Core behavior, where media attached to posts doesn’t get deleted when the posts get deleted in case you want to re-use them in other posts.

    Plugin Author Konstantin Obenland

    (@obenland)

    1 month, 1 week ago

    “Good” news, I can reproduce the outbox error with the archive you shared with me. The import file gets unzipped, but for some reasons the files get lost, so I’ll see if I can track down the cause of that.

    Thread Starter TheVoidTLMB

    (@thevoidtlmb)

    1 month, 1 week ago

    Ha, “great”! 🙂

    Plugin Author Konstantin Obenland

    (@obenland)

    1 month, 1 week ago

    Got it. I submitted a fix in https://github.com/Automattic/wordpress-activitypub/pull/2581.

    Thread Starter TheVoidTLMB

    (@thevoidtlmb)

    1 month, 1 week ago

    Excellent, I’ll try again when 7.8 comes out then. 🙂

    Thread Starter TheVoidTLMB

    (@thevoidtlmb)

    1 month, 1 week ago

    Another update:

    On my main blog, I updated to WP 6.9, and AP 7.7. I updated a few other plugins and uninstalled an unactivated one. I thought I might as well try to import file #1 again…

    “Skipped posts:
    https://mastodon.online/users/TheVoidTLMB/statuses/109297913396719794
    https://mastodon.online/users/TheVoidTLMB/statuses/109302851299775723
    Imported 17 posts.
    All done. Have fun!”

    Those two skipped posts are the ones that were previously the only two successfully imported ones. I deleted them before a later attempt – I wonder if that’s be relevant, since you said “the importer skips already imported posts”. Well, they’re still in my Bin (ah – that’ll be it, right? They still are technically “there”, so they were skipped, I assume?) so I can just Restore them.

    Done – they came back as Drafts, and I republished them one at a time. (The fediverse setting was pre-set to Do Not Federate – nice.)

    Although that’s a success, it’s slightly frustrating not to know why the database error went away, but I’ll take it!

    Archive #1 sorted, hurrah. Archive #2 to wait until the json error fix is pushed through.

    Thanks for all your continued help! @obenland

    Plugin Author Konstantin Obenland

    (@obenland)

    1 month, 1 week ago

    they’re still in my Bin (ah – that’ll be it, right? They still are technically “there”, so they were skipped, I assume?) so I can just Restore them.

    Yes, that’s correct.

    Archive #2 to wait until the json error fix is pushed through.

    That should be today or tomorrow. The PR was merged and we’ll ship a patch release soon.

    Plugin Author Konstantin Obenland

    (@obenland)

    1 month, 1 week ago

    I found another hiccup with the 72 MB export file, in that the media attachment URLs are prefixed with an unexpected path string which breaks the media file import.

    To fix that you could open the outbox.json file (in the extracted folder) in a file editor and replace /v1/AUTH_011f6e315d3744d498d93f6fa0d9b5ee/qotoorg with an empty string. Or run the following terminal (replace /path/to/outbox.json with the actual path):

    sed -i '' 's|/v1/AUTH_011f6e315d3744d498d93f6fa0d9b5ee/qotoorg||g' /path/to/outbox.json

    qoto.org seems to run quite an outdated version of Mastodon, v3.5.19.

    Thread Starter TheVoidTLMB

    (@thevoidtlmb)

    1 month, 1 week ago

    Oh yes, I hadn’t noticed the media weren’t imported in my test run. Thanks for that, I’ll give it a go.

    Qoto being stuck on 3.5 was the main reason I left. I later saw some posts about them doing some updates, but if their About page is to be believed, they’re still on 3.5.19, yeah. Hmm.

Viewing 15 replies - 16 through 30 (of 36 total)
1 2 3

You must be logged in to reply to this topic.