Bug in the main.js

tempuser09112023

Pretty print main.js, at line 15163, parent_id: e.parent_id?.substring(3) || t, should be changed to parent_id: e.parent_id ? He(e.parent_id) : t,

pullpush-actual

Thanks for the report, which app is that?

tempuser09112023

pullpush-actual It's the main.js of undelete.pullpush.io

pullpush-actual

If you feel adventurous, it is open source. You can do PR here: https://github.com/pullpush-io/undelete

tempuser09112023

pullpush-actual
Code of undelete: https://github.com/pullpush-io/undelete/blob/main_pub/src/api/pushshift/index.js#L142
const exitEarly = !callback(comments.map(c => ({ ...c, parent_id: c.parent_id?.substring(3) || threadID, link_id: c.link_id?.substring(3) || threadID })))
Code of unddit: https://github.com/gurnec/removeddit/blob/master/src/api/pushshift/index.js#L152
const exitEarly = !callback(comments.map(c => ({ ...c, parent_id: c.parent_id ? toBase36(c.parent_id) : threadID, link_id: c.link_id?.substring(3) || threadID })))
The bug is clear, for parent_id of comments, should add a toBase36() to make sure the format is correct.
Reason could be found at: https://github.com/pullpush-io/undelete/blob/main_pub/src/api/pushshift/index.js#L78
// Starting sometime around May/6/2022, queries using the 'ids' parameter // started returning decimal IDs for the '_id' members instead of fullnames. const toBase36 = id => { if (!id) return id if (typeof id == 'number') return id.toString(36) else return id[2] == '_' ? id.substring(3) : id }

pullpush-actual

Thanks. Can you give me an id where id is returned as a number? It might happen as April - Sept data was partially scraped from pushshift.

tempuser09112023

pullpush-actual
Under the post: https://undelete.pullpush.io/r/wallstreetbets/comments/1453fmn/the_house_always_wins/ , the script will fetch https://api.pullpush.io/reddit/comment/search/?fields=author,body,created_utc,id,link_id,parent_id,retrieved_on,retrieved_utc,score,subreddit&metadata=true&size=100&sort=asc&link_id=1453fmn , and you can find that the comments jnjf1vy, jnjgb0h, jnjgy0p, jnjljfm, jnjqr4s, jnjr10o have a parent_id returned as a decimal number

tempuser09112023

pullpush-actual Another problem is that, all those comments are recored 2 times. In the search result, you can find out that the 1st record of each comment has a correct parent_id in base36, but their body are already been removed or deleted; the 2nd record of each comment has the correct comment body, which shows the content before removal, but their parent_id are in decimal format.

pullpush-actual

tempuser09112023

Yes, this is will be addressed as soon as I get some time to do a round of maintenance. The code needs to decide which is "the correct" version. This isn't as simple as selecting the first or the last since approve-only subs start as deleted and obviously content can be removed later.

tempuser09112023

pullpush-actual Well, you don't really need to judge which is the "correct" version, you can simply mix them, and create a "correct" one. What I mean is that, since the only useful values for a comment is the author and body, and, they are also the only values which could change in different versions, (for the same comment, no matter which version, they always have the same id, created_utc etc), so you can just extract the correct author and body, then "create" a comment with the rest info of any version.

For the author, it only has 2 possible values, normal or [deleted], so if any version has a normal value, take it.

For the body, there are more choices, normal, [deleted], [removed], [ Removed by Reddit ], but all 3 last values are useless, no matter if the comments is really removed, or it's just someone who has literally typed "[removed]", they are worthless anyway, so you can also simply pick a normal value from any version.