Wayback Machine: MDL Archive

Discussion in 'Announcements' started by MrMagic, Nov 25, 2016.

  1. MrMagic

    MrMagic MDL Guru

    Feb 13, 2012
    6,015
    4,148
    210
    Yea, not sure how it's doing things, definitely not one page at a time

    Going to take forever though, currently 4200/584000
     
  2. MrMagic

    MrMagic MDL Guru

    Feb 13, 2012
    6,015
    4,148
    210
    Not sure this method is going to work, in my estimation at this speed it will not complete for 139 hours, almost 6 days

    Doesn't look like it will speed up any either
     
  3. ancestor(v)

    ancestor(v) Admin
    Staff Member

    Jun 26, 2007
    2,974
    6,051
    90
    Don't stress it. I'm on it as well. :tea:
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. MrMagic

    MrMagic MDL Guru

    Feb 13, 2012
    6,015
    4,148
    210
    Ah good, how fast is yours going?

    I'm averaging 1-2 per second
     
  5. CaptainKirk1966

    CaptainKirk1966 Former MDL Guru

    Oct 31, 2009
    2,549
    1,377
    90

    It seems a clean, even if slow, method. I finally got the pages I downloaded looking much like the originals. I'm surely not going to try to download the entire site this way.
     
  6. MrMagic

    MrMagic MDL Guru

    Feb 13, 2012
    6,015
    4,148
    210
    Very slow, but if it's the only option, I guess a weeks worth of work isn't too much to get the site back as it was
     
  7. Yen

    Yen Admin
    Staff Member

    May 6, 2007
    13,081
    13,980
    340
    I wonder what will be downloaded. The forums are actually a sql db
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. Tito

    Tito Super Mod / Adviser
    Staff Member

    Nov 30, 2009
    18,681
    18,589
    340
    AFAIK, it downloads each page as static HTML one, which are originally dynamically generated by vBulletin from the DB.
     
  9. CaptainKirk1966

    CaptainKirk1966 Former MDL Guru

    Oct 31, 2009
    2,549
    1,377
    90

    That is correct. A hierarchy of static web pages.

    .
    ./websites
    ./websites/forums.mydigitallife.net
    ./websites/forums.mydigitallife.net/image.php?u=292305&type=sigpic&dateline=1435916606
    ./websites/forums.mydigitallife.net/image.php?u=416188&dateline=1391784116&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=151&dateline=1400925404&type=thumb
    ./websites/forums.mydigitallife.net/start.html
    ./websites/forums.mydigitallife.net/image.php?u=424895&dateline=1396368096&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=308117&dateline=1434637536&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=78164&dateline=1255626065
    ./websites/forums.mydigitallife.net/image.php?u=113479&dateline=1458073541&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=383491&dateline=1460939820&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=113479&dateline=1458073541
    ./websites/forums.mydigitallife.net/image.php?u=75821&dateline=1256950839&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=104688&dateline=1457482375&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=383491&dateline=1460939820
    ./websites/forums.mydigitallife.net/vb4_style
    ./websites/forums.mydigitallife.net/vb4_style/smilies
    ./websites/forums.mydigitallife.net/vb4_style/smilies/thumbsup.gif
    ./websites/forums.mydigitallife.net/vb4_style/smilies/rip.gif
    ./websites/forums.mydigitallife.net/faq.php
    ./websites/forums.mydigitallife.net/image.php?u=403654&dateline=1442019749
    ./websites/forums.mydigitallife.net/image.php?u=416188&dateline=1391784116&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=424895&dateline=1396368096&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=218536&dateline=1423457849
    ./websites/forums.mydigitallife.net/image.php?u=465989&dateline=1442061127
    ./websites/forums.mydigitallife.net/image.php?u=373819&dateline=1443990618&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=292305&dateline=1435914555
    ./websites/forums.mydigitallife.net/image.php?u=105&dateline=1441592476
    ./websites/forums.mydigitallife.net/image.php?u=292305&type=sigpic&dateline=1435916606
    ./websites/forums.mydigitallife.net/image.php?u=1069&dateline=1302090631
    ./websites/forums.mydigitallife.net/image.php?u=16886&dateline=1296142680
    ./websites/forums.mydigitallife.net/image.php?u=98324&dateline=1448234475
    ./websites/forums.mydigitallife.net/image.php?u=80287&dateline=1305515589
    ./websites/forums.mydigitallife.net/attachment.php?attachmentid=32379&stc=1
    ./websites/forums.mydigitallife.net/image.php?u=411048&dateline=1389553690
    ./websites/forums.mydigitallife.net/image.php?u=789783&dateline=1478949864&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=268073&dateline=1461594161&type=thumb
    ./websites/forums.mydigitallife.net/image.php?u=290757&dateline=1437931331
    ./websites/forums.mydigitallife.net/image.php?u=248996&dateline=1472370929&type=thumb
    ./websites/forums.mydigitallife.net/threads
    ./websites/forums.mydigitallife.net/threads/63471-KMS_VL_ALL-Online-Offline-KMS-Activator
    ./websites/forums.mydigitallife.net/threads/63471-KMS_VL_ALL-Online-Offline-KMS-Activator/index.html
    ./websites/forums.mydigitallife.net/threads/58464-Windows-Loader-Download?
    ./websites/forums.mydigitallife.net/threads/58464-Windows-Loader-Download?/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1807
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1807/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page641
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page641/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1927
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1927/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page11?
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page11?/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1423?
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1423?/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page514
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page514/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1455
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1455/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1886?p=935638&viewfull=1#post935638
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1886?p=935638&viewfull=1#post935638/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2122?p=1140628&viewfull=1#post1140628
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2122?p=1140628&viewfull=1#post1140628/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2074?p=1101419&viewfull=1#post1101419
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2074?p=1101419&viewfull=1#post1101419/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1474
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1474/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2185?p=1208567
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2185?p=1208567/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1997?p=984687&viewfull=1#post984687
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1997?p=984687&viewfull=1#post984687/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1673
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1673/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1933?p=951908#post951908
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1933?p=951908#post951908/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1806?
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1806?/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2119
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2119/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2128
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2128/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1968
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1968/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1243
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1243/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1660
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1660/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2111
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2111/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page101
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page101/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page516
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page516/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1510
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page1510/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2051?
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page2051?/index.html
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page602
    ./websites/forums.mydigitallife.net/threads/24902-Windows-Loader-Support-and-chat/page602/index.html
     
  10. Yen

    Yen Admin
    Staff Member

    May 6, 2007
    13,081
    13,980
    340
    I think we try to get the full one either way. Ancestor(v) is on it. It can be used if important threads are missing in the latest backup. Not sure if it could be automated to implement into the db easily.

    I thank you for your efforts. :)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. MrMagic

    MrMagic MDL Guru

    Feb 13, 2012
    6,015
    4,148
    210
    So there will still be a fair amount of manual work to be done once it has downloaded everything

    Main thing is we get the content back I suppose
     
  12. T-S

    T-S MDL Guru

    Dec 14, 2012
    3,984
    1,331
    120

    Obviously the forum is a db. A single file or few files large hundreds of MB.

    Downloading the static pages as they are can be useful just to recover the links inside the threads and little more. Unless someone writes an import script to automate somewhat the process.

    The sad truth is that what we need is a db copy. 3 months old, doesn't matter. The problem are not the recent threads, the problem is the old knowledge.

    Especially on thread like the WMC one, which were unique over the whole internet.o_O
     
  13. GOD666

    GOD666 MDL Expert

    Aug 1, 2015
    1,958
    2,061
    60
    This is fact. "Way back time machine" only makes a static HTML image (a copy of the site of how it would look in a user's browser). It cannot be imported directly back into MDL through this method.

    Though it could be used to offer an archived version somewhere.