Recovering auto-pruned posts in PHPbb forums...from SQL file backups, instead of the internal PHPbb tool

I've been using PHPbb since V2 It's an excellent tool if you want to create a standard forum without a fuss in record time. I recon that I«ve been off for a while since V2 , but recently i had the need to create 2 sites that, according to specs where just simple forums, so I returned to PHPbb and found my self on V3.
During the rush of putting the sites up A.S.A.P /BTW with install, full config, sample posts, creating emails invite emails, changing the email templates and code to include registration custom fields and template design and implementation of template design with google adds code embed, it took-me a total of 2 hours to get everything done. OOhh and it's 2 hours for the 2 forums, not 2 hours each! Not you have to respect that. Free code that saves you countless ours of coding and templating and configuring time. And mind that I was "rusty" (I didn't PHPbb since V2 and a lot has changed). This, is productivity that HAS to be appreciated: THANKS PHPbb team.

I decided to post this article after finding some misleading information on the PHPbb forums stating (and I quote): "to script to do that automatically has been created yet", "it's just too complex, better restore the entire database".

OK, you have instructions on how to restore and backup de database using PHPbb internal tools (makes sense as it IS the supported feature) but let's just assume that, just like me, you have much more to deal with and you configure your servers to backup automatically using cpanel, for instance.
That means that you will not be using the internal PHPbb backup tool and as a result you'll be able to restore or backup at MySQL (or your db of choice) files level. That is also a good way to work right until the point where you find out that you left "autoPrune = yes" on one of your forums.
You see Auto  Prune is a nice admin tool. Let's just say for instance that you create a forum for improvement requests... you want to have people writing there, but you don't want those posts to stay there for ever. They just loose sense when either you implement, or reject the improvements users request.
However if you just enable it in one of your other content forums, and please mind that by default it is set to prune posts over 7 days with no activity, you might just find your self visiting a ghost forums (especially if you are still starting it and you have few people writing on it.

Now picture you forget that setting on, and also use Cpanel to backup. Well than I guess you'll find your way into this post (or the other forums posts I've quoted") sooner or later.

It's easy to restore your pruned posts if you have an SQL for for your database restore.
Just open your administration tool (PHPMyadmin for instance). At the same time, open your SQL restore file on a text editor and then browse the tables and the respective restore code to check what has been deleted and out of those what do you want to restore.
Then use an SQL tool to run commands on your server and copy-past from the backup, file, clean up unwanted lines and run the command.
It's that simple. The ONLY thing you absolutely need to understand is that there are several tables that have post information and indexing to forums and topics.
So you'll have to perform this procedure in the following tables (and add the full lines to the missing table):
   - PHPbb_posts
   - PHPbb_topics
   - PHPbb_topics_posted

This will render your posts back on to your forum. However, you you also want the forum statistics and last post and poster up-to-date, you need to restore a part of your forums table. So inside PHPbb_forums, find the following fields and restore them from your backup (probably using either an update command or simple copy-paste into location, depending on just how much you need to restore)
   - forum_posts
   - forum_topics
   - forum_topics_real
   - forum_last_post_id
   - forum_last_poster_id
   - forum_last_post_subject
   - forum_last_post_time
   - forum_last_poster_name
   - forum_las_poster_colour

When you finish this procedure, now only you've recovered your posts and topics, you have also restored those statistics indicators on the forum's front page that will point your users to the most active topics.

That's it, that's all. Simple and not by all means in need of a script or a complicated procedure.

Recover files from a VMFS virtual volume inside a iSCSI virtual volume

The problem:
Some time ago, I had a real bad RAID failure. Since I use my VMWARE ESXi hardware just to boot from a pen drive and connect to my QNAP NAS via iSCSI, a raid failure means the RAID degrades and enters read-only mode. Linux Virtual Machines keep working with errors on some packages, but windows virtual machines freeze in a couple of seconds.

The Solution:
Since I'm not rich and buying a full NAS system to copy from the read-only to the new one is out of question, I brought 2 2tb USB drives, connected them to my second NAS and moved the files out to the drives making room to move files from one NAS to another.
Now until this part it's all very simple; grab NAS02, make a desktop terminal connect into both iSCSI target on NAS01 and NFS share on NAS02 and move files from one to the other...and you start by installing open-iscsi.
HOWEVER, vmware hypervisors use a specially designed file system called VMFS, so mounting your iSCSI volumes will get you nowhere.
That's when vmfs tools get into action. Install the vmfs tools package and then all you have to do is mount the drives. It's a crazy opp, but it works. Basically, You'll be mounting the virtual iSCSI volume into a virtual directory, then the VMFS virtual volume that the iSCSI virtual volume contains into another virtual directory.
Better have Gigabit Ethernet for the next part: After this mount the NFS from the second NAS, and copy everything...for a couple of days, depending on your network speed :S

Sources I used to study:

• http://planetvm.net/blog/?p=1592
• http://www.cyberciti.biz/faq/howto-setup-debian-ubuntu-linux-iscsi-initiator/
• http://www.howtoforge.com/using-iscsi-on-ubuntu-9.04-initiator-and-target

How did I mount the iSCSI, then then VMFS virtual volumes in my linux desktop? here goes the command LOG:
administrator@RWS01:~$ su
Password:"SECRET"

root@RWS01:/home/administrator# /etc/init.d/open-iscsi start
 * Setting up iSCSI targets                                              [ OK ]

root@RWS01:/home/administrator# mount 10.0.101.246:/RECOVER /iscsi/rec

root@RWS01:/home/administrator# /etc/init.d/open-iscsi restart
 * Disconnecting iSCSI targets                                           [ OK ]
 * Stopping iSCSI initiator service                                      [ OK ]
 * Starting iSCSI initiator service iscsid                               [ OK ]
 * Setting up iSCSI targets                                              [ OK ]

root@RWS01:/home/administrator# iscsiadm --mode discovery --type sendtargets --portal 10.0.101.247
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
10.0.101.247:3260,1 iqn.2004-04.com.qnap:ts-639:iscsi.vmwareisdata.bd99c2
10.0.101.247:3260,1 iqn.2004-04.com.qnap:ts-639:iscsi.vmwareboots.bd99c2

root@RWS01:/home/administrator# /etc/init.d/open-iscsi restart
 * Disconnecting iSCSI targets                                           [ OK ]
 * Stopping iSCSI initiator service                                      [ OK ]
 * Starting iSCSI initiator service iscsid                               [ OK ]
 * Setting up iSCSI targets                                              [ OK ]

root@RWS01:/home/administrator# tail -f /var/log/messages
Nov 23 01:44:01 RWS01 kernel: [  391.203611] sd 22:0:0:0: reservation conflict
Nov 23 01:44:01 RWS01 kernel: [  391.203638] sd 22:0:0:0: [sdc] READ CAPACITY failed
Nov 23 01:44:01 RWS01 kernel: [  391.203640] sd 22:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Nov 23 01:44:01 RWS01 kernel: [  391.203643] sd 22:0:0:0: [sdc] Sense not available.
Nov 23 01:44:01 RWS01 kernel: [  391.206527] sd 22:0:0:0: reservation conflict
Nov 23 01:44:01 RWS01 kernel: [  391.207253] sd 22:0:0:0: reservation conflict
Nov 23 01:44:01 RWS01 kernel: [  391.208968] sd 22:0:0:0: reservation conflict
Nov 23 01:44:01 RWS01 kernel: [  391.209013] sd 22:0:0:0: [sdc] Test WP failed, assume Write Enabled
Nov 23 01:44:01 RWS01 kernel: [  391.211881] sd 22:0:0:0: reservation conflict
Nov 23 01:44:01 RWS01 kernel: [  391.211903] sd 22:0:0:0: [sdc] Attached SCSI disk
tail -f /var/log/messages
^C

root@RWS01:/home/administrator# mkdir /iscsimount

root@RWS01:/home/administrator# mount /dev/sdb1 /iscsimount
mount: you must specify the filesystem type

root@RWS01:/home/administrator# sudo vmfs-fuse /dev/sdb1 /iscsi/4

root@RWS01:/home/administrator# sudo ls /iscsi/4 -alh
total 4.0K
drwxr-xr-t 9 root root 2.0K 2010-06-05 21:06 .
drwxr-xr-x 9 root root 4.0K 2010-11-22 20:18 ..
-r-------- 1 root root 2.5M 2010-02-28 18:44 .fbb.sf
-r-------- 1 root root  61M 2010-02-28 18:44 .fdc.sf
-r-------- 1 root root 244M 2010-02-28 18:44 .pbc.sf
-r-------- 1 root root 249M 2010-02-28 18:44 .sbc.sf
-r-------- 1 root root 4.0M 2010-02-28 18:44 .vh.sf
drwxr-xr-x 2 root root  560 2010-11-14 18:19 VNAS01_BackupServer_UNX
drwxr-xr-x 2 root root 2.5K 2010-11-15 19:18 VSRV06_DomainServer_2k8
drwxr-xr-x 2 root root 6.6K 2010-11-15 19:15 VSRV07_LTSDomainServer_2k8
drwxr-xr-x 2 root root 3.1K 2010-10-19 01:31 VSRV07_Sharepoint_2k8
drwxr-xr-x 2 root root 2.8K 2010-11-15 19:07 VSRV08_WebServer_2k8_R2_64
drwxr-xr-x 2 root root 2.5K 2010-08-07 03:20 VWKS03_Private_XP32
drwxr-xr-x 2 root root 3.5K 2010-08-23 16:45 VWKS04_Testbench1_XP32

root@RWS01:/home/administrator# sudo vmfs-fuse /dev/sdd1 /iscsi/5

root@RWS01:/home/administrator# sudo ls /iscsi/5 -alh
total 4.0K
drwxr-xr-t 11 root root 2.2K 2010-03-06 04:29 .
drwxr-xr-x  9 root root 4.0K 2010-11-22 20:18 ..
-r--------  1 root root 2.5M 2010-02-28 04:20 .fbb.sf
-r--------  1 root root  61M 2010-02-28 04:20 .fdc.sf
-r--------  1 root root 244M 2010-02-28 04:20 .pbc.sf
-r--------  1 root root 249M 2010-02-28 04:20 .sbc.sf
-r--------  1 root root 4.0M 2010-02-28 04:20 .vh.sf
drwxr-xr-x  2 root root 2.6K 2010-11-15 19:22 VRV01_DEVServer_2k3
drwxr-xr-x  2 root root  280 2010-02-28 16:32 VSRV01_Development_2k3
drwxr-xr-x  2 root root 2.9K 2010-11-15 19:22 VSRV02_LNX_VPN_WEB_DB
drwxr-xr-x  2 root root  280 2010-02-28 16:16 VSRV02_VPN_MYSQL_APACHE
drwxr-xr-x  2 root root 2.8K 2010-11-15 19:16 VSRV03_DBServer_2k3
drwxr-xr-x  2 root root 3.2K 2010-11-15 19:14 VSRV04_WEBServer_2k8
drwxr-xr-x  2 root root 2.6K 2010-11-15 19:22 VSRV05_DBServer_LNX
drwxr-xr-x  2 root root 2.8K 2010-08-07 03:22 VWKS01_Downloader_XP32
drwxr-xr-x  2 root root 2.5K 2010-11-15 19:22 VWKS02_Monitor

root@RWS01:/home/administrator#
THAT's COPY TIME NOW

to RAID or not RAID.... what was the question again?

Hi everyone,

In my basement setup I've got a nice virtualized setup. A single VMWare ESXi4 server boots form a 4gb pen drive and then connects via iSCSI.

Supporting the connections is a SMC Gb switch.

Supporting data are 2 QNAP NAS. A 6 drives 639 and a 4 drives 459 pro.

Each one is fully populated with 2Tb Samsung Hardrives and WesternDigital.

The 639 is configured with a raid5 of 4x 2Tb 7200prm 32mb cache Samsung HD and 2 separate 1Tb drives

The 459 is configured with a raid5 of 4x 2Tb 7200prm 32mb cache Samsung HD 

On top of this the 639 has:

 - 1 320Gb USB

 - 3 1.5Tb USB

 - 1 750Gb USB
 - 1 1,5Tb eSATA
 - 1 2Tb eSATA

The 459 has:

 - 4 1Tb USB

 - 1 1,5Tb eSATA
 - 1 2Tb eSATA

The event:

Power outage over the 10minute UPS threshold, then another one while the NAS was booting up again. Portuguese power company is probably the worse in existence. I pay 200€/month for my house power supply (yet 200€/month) and I still get frequent power outages and every time I file a complaint... it's someone else's fault but theirs).

The consequences:

The 459 NAS hold out fine as is consumes less power so is never shutdown between outages.

The 639 came up after 2 abrupt shutdowns with a 1 disk failure. The hard drive was fine but it just didn't rebuild. I then changed the hard drive and the NAS declared it as a spare and switched the mode to degraded and made everything read-only.

The solution:

After having the NAS in degraded-readonly mode, all I could do was to backup my data and rebuild the RAID from scratch.

The problem:

1 - 4,7Tb of data are not an easy backup.

2 - most of it was in a iSCSI virtual volume.

3 - The data inside the virtual volume is formated in a VMFS partition.

So how do you solve this:

1 - Go buy 2x 2Tb USB HD

2 - Backup everything into the existing USB HD
3 - Clear the RAID, test the drives independently, replace if the drive actually has problems (some QNAP reports aren't fully accurate on drive problems).
4 - Re-create the RAID
5 - Move the iSCSI VMFS drive contents to the now clean RAID.

6 - The most important part: setup full constant replica's between the two devices.

DID QNAP SUPPORT HELP?

Not quite. The first EMail was an obvious... backup your data and do-it all over again. After a twisted reply from my side ("is this what you think of how a RAID works? a drive fails the new one doesn't rebuild and it's a backup and do-it all over again time? should I start looking for an alternative solution and return my QNAPS?") Then the reply was different. They asked for a link so they could remotely access and analyse the problem. It happened however 1 week after the event... and that's everything but a good support. The conclusion was that I had to backup the NAS and re-do...jezz thanks QNAP, that's really pro!

My Setup Today:
NAS01                                                                      NAS02
  RAID5 (SATA2,3,4,5)                                             RAID5(SATA1,2,3,4)
      VMachinesDepot1 <-------------------------------------> ReplicaOfVmachinesDepot1
      ReplicaOfVmachinesDepot2 <------------------------> VMachinesDepot2
      LocalWorkShare <---------------------------------------> ReplicaOfLocalWorkShare
      ReplicaOfVPNWork <----------------------------------> VPNWork
      FamilyMedia <--------------------------------------------> ReplicaOfFamilyMedia
  eSATA1                                                                     eSATA1
      TVSeries                                                                    Movies
  USB1                                                                         USB1
      BackupsFromClients                                                  BackupsFromServers
  USB2                                                                         USB2
      KidsMovies                                                                MusicAndVideos
  eSATA2                                                                     eSATA2
      Downloads                                                                 Temp
  USB3                                                                         USB3
      Software                                                                      Library
  USB4
      Tutorials
  SATADisk1                                                              USB4
      CompanyWork <------------------------------------------->ReplicaCompanyWork
  SATADisk6
    OLDProjectsBackup

So in conclusion:
 Use the RAID5 for Safety...and don't trust it, so replicate to the other RAID.
 So in short... today Hardware RAID is actually a Software RAID with a specific hardware appliance and controlled software packaging and configuration. QNAP (and the other players like it) can sell systems that look professional, but in truth they are not different from a standard PC with a SATA card, running linux and MDADM, with a simple interface and a fancy hotplug drawer system.
 Support is as "pro" as the standard product is.
Not a bad product, especially price-wise, and that's the line you should think of. It's a great solution, not because it costs 1/4 or less, but rather because that 1/4 price tag allows you to buy twice as much as you intended and have full redundancy. So in truth it's NOT as cheap as it claims to be, but taking some simple precautions it can be 1/2 of what a pro system would cost.