the great ubuntu NAS project

Here we go folks.  Complete, yet half-baked documentation on how to setup your very own home NAS using free Linux software.  I know this has been written about before, and in fact I have pulled from many fine sources (see my acknowledgement below).  This is documented here mainly so I don’t forget how I did this.  It’s not really for you – sorry internet.  Let’s get started:
Continue reading “the great ubuntu NAS project”

switching from FreeNAS to Ubuntu Server

This is basically part two of my previous post about troubles in NAS-land.  That experience got me thinking about switching up my NAS solution.  Here’s why:

  1. FreeNAS v0.7.x is based on FreeBSD which is a fairly foreign OS to me.  Everything, I mean everything, that I do in that OS is a complete and total learning experience.  While it’s easy to setup initially, debugging is a chore.  I’m much more comfortable in either a Windows or Linux environment.
  2. FreeNAS v0.7.x has basically been EOL’d and is now in “legacy” support.  That leaves me wondering about security and stability issues when upgrading to new hardware (see my previous post).
  3. FreeNAS v0.7.x transfer speeds have been shown to be a bit slower than (reference needed) other solutions for SAMBA or NFS.
  4. FreeNAS v0.7.x uses UFS (Unix File System).  Yeah… I don’t know either.  Mounting EXT4 or NTFS is not an option, but would be extremely handy when swapping data around.
  5. FreeNAS v0.7.x has questionable USB drivers which seemed to hang on my system when doing large RSYNC or file copy sessions.  Bad.  Also… slooow.
  6. FreeNAS v0.7.x has PHP4 and Lighttp instead of PHP5 and Apache2.  There were a bunch of cool web applications that I wanted to install on FreeNAS but didn’t because I didn’t want to hassle with installing PHP5 and Apache and screwing up the Web UI.  It looks messy and difficult.

Given that I’m comfortable with Linux (Ubuntu specifically), I was eager to see if I could make the switch to something more Linux’y, and still have the nice features and flexibility of FreeNAS.  Here was what I wanted:

  1. I want to run the OS from a USB flash drive.  Well ideally it would be nice to run it from an SSD, but I’m not made of money.
  2. I want a comparable Web UI to FreeNAS (huge selling point).  Needs to have the ability to manage shares, services, show me uptime, transfer rate, etc.
  3. RAID 1 mirror capability is a must.  I have data that I want protected from a drive failure.  Redundancy is key to my strategy.
  4. Services like SMB/CIFS, SSH, NFS, FTP, etc. are important to access the data.
  5. File system flexibility.  I’d like the capability to attach new storage in different formats.  Linux gives me that flexibility.  EXT4 gives me the performance and the protection from issues like power-outages that killed my previous NAS solution dead.
  6. Remote Bittorrent.  Gotta have it.
  7. Subsonic (hell yes)
  8. Exandability.  I want to install my own web apps, servers like Plex Server, Tonido, OwnCloud, and a host others that would have just been a pain in FreeNAS.
  9. Good hardware support and regular security updates.
  10. Good support community.

Here was my final verdict:

  1. Yes, you can run Ubuntu Server from a USB flash drive.  Some folks have pointed out that USB flash drives may run the risk of wear-out and warn to not install an OS on a USB flash drive.  You can minimize the amount of writes that are targeted at the flash device by removing the swap partition (make sure you have enough system memory for this).  Others say that you should probably never run into a problem with a good quality USB device.
  2. I started to look for alternative Web UI’s for server administration and I stumbled across a few.  I was most impressed by Ajenti (http://ajenti.org), however it’s still in an very early stage of development.  I’ll probably go with this, but Webmin appears to be pretty widely used UI and I may fall back on this to get more functionality.
  3. Yup, I’ve done a lot with RAID Mirroring on Ubuntu in the past.  I’ll just dig up this old post (raiding the archives) and implement a mirror as I did before with MDADM.  Easy.
  4. Check.  SMB/CIFS, SSH, NFS, FTP are all a snap to install on ubuntu.  Just pull from the repositories, configure and off we go.  Ajenti has some capability to manage these services from the Web UI as well.
  5. Yes again.  Flexibility abounds.  I will choose EXT4 thank you very much.
  6. Same as FreeNAS, I’ll pull in the latest Transmission Deamon.
  7. Yup – and easier to install, upgrade and manage on Ubuntu.  It’s a deb package so I won’t have to much other than configure it once it’s installed.
  8. Resounding yes.  I will have full flexibility to install the latest PHP, Apache, MySQL, etc… this means I can go crazy installing other fun projects on the NAS in the future.
  9. Yes and yes.
  10. Yes – while FreeNAS has a great community, Ubuntu has a bigger user base and a bigger support community.

So that’s it.  I’ve made up my mind.   Ubuntu Server here we come.   It’s going to be quite a project to get this initially setup, but I think it will be an enriching and useful project.  It’ll open my home server up to new capabilities and hopefully enhanced performance.   FreeNAS is great, but it’s sort of like riding a bike with the training wheels on.  It’s time to ride on two wheels now.

P.S. I should also mention that during my research, I ran across OpenMediaVault (http://openmediavault.org/).   The original FreeNAS project forked into two projects.  The first fork was FreeNAS 8.0 based on FreeBSD8, which is aimed at higher-end server equipment.   I avoided this version because it is still in it’s infancy, requires a kind of ridiculous amount of power and it’s still based on FreeBSD.  I recently discovered that the second fork is actually a Debian Linux based fork aimed more at the home NAS market.   Now they have me second guessing things.  Since it’s based on debian, perhaps this would be the easier route… naw.   

troubles in NAS-land

If you’ve followed my blog in the past, you probably have noticed that I have been a big fan and user of FreeNAS for my home NAS needs.  FreeNAS is a great project and quite excellent for those who want a basic, easy to use, polished NAS OS for free.  I can’t speak enough to how good FreeNAS is.  However, recently, I’ve had some challenges that caused me to re-think my home NAS strategy.

I should probably follow the old saying that “if it ain’t broke, don’t fix it”.  In this case, I totally disregarded that wise advice and “fixed” with wild abandon.  With my current NAS operating relatively glitch free for over a year on FreeNAS, of course it was time for some upgrading and tinkering.

It started out as an idea for offsite backup, which involved installing duplicity (which turned out to not be that hard), which presented some challenges.  More on that later.  I was also doing some large file transfer across my home network, which is gigabit but was struggling to achieve a decent transfer rate.  I decided after some research that my hardware was holding me back, it was probably my 2-3 year old chipset bottlenecking transfer speeds.  Fortunately, I had just decommissioned another system in my home with considerably greater horsepower.  That’d be great to replace this aging slow NAS system.  So backed up my data on the NAS to some old HD’s laying around the house (slowly… across the network), and switched to the new hardware.  Upgraded FreeNAS in the process to the latest stable v0.7.2.  Restored my config.  Good so far.

Then, my FreeNAS box started rebooting randomly.  Worse, it rebooted and hung while we were away in Hawaii for two weeks.  Nothing I hate worse than having no access to my tunes, let alone other data, and having a broken system that I can’t do anything about, a thousand miles away.  At first I thought I was getting power-outages.  But I discovered later that it was nothing to do with our power at the house.   After a look through syslogs, and some internet searching I found my problem.  Corruption.

I was able to run fsck (file system check) on the RAID array and get the darn think back serviceable again… but the reboots continued.   What… the… hell.  I started to go through my mental checklist of what could be going wrong with the system.  Why the crazy random reboots?  And absolutely no warnings in the log files, no errors, no kernel panic, nothing.  Had to be something with the motherboard, cpu, memory or power-supply or something.  After a while I decided to replace the motherboard first… which forced me to replace the memory.  Two more variables removed.  Still, reboots.  Finally I replaced the power-supply.  Bingo – problem solved.   Why did I even bother to do an upgrade in the first place?!

Well, at least we are stable for now.  Stay tuned for part two… switching to Ubuntu Server from FreeNAS.

accessing your files on freenas from anywhere

You may remember that I had posted about setting up a remote file server (via the web interface), a little while ago. Well I just stumbled across this post which talks about using secure FTP via the ssh tunnel. It’s dead simple to setup and is actually quite an elegant solution.  It does require FTP software that supports SFTP, so it’s not quite as user friendly or configurable, but it’s great for power-users.  I think I audibly said, “duh!” when I stumbled upon this.  Also, it’s nice having the SSH capability exposed externally.  Note that the tutorial is for FreeNAS 8, but is totally applicable to FreeNAS 7 (if you’re like me an are sticking to the legacy server software).

And regarding security, I would definitely use a different port than the standard SSH port, at least externally. Otherwise you’ll get clobbered with traffic from hackers attempting to connect to your server for not so nice reasons.  You may want to consider using a keyfile as well – which makes it less convenient, but much more secure.

a year of results

The year 2011 just flew by.  I’m happy to report that after a year of dedication to a new weight loss strategy, I actually got some pretty good results.  Let’s re-wind a little.  Way back in April of 2011, I briefly touched on my efforts to lose some weight over the course of the year.  Back then, I was talking about this new fad diet and how I had little motivation to exercise.  At the beginning of the year, I started the slow carb diet (inspiration from Tim Ferriss) and started meticulously documenting my weight and body fat percentage daily (also documented my measurements).  I’m proud to say I kept it all up over the entire year.

Only one thing has changed since then. I’m still on that fad diet and I still haven’t found my motivation to exercise.   However, my weight has dropped pretty drastically despite my sedentary nature.  I dropped over 40 pounds during the year to be more accurate.  That’s more than three (3) average sized bowling balls worth of extra weight that I was carrying with me everywhere, putting stress on my bones, my heart and my mind.

This is great and all, but there is another good 10 or so pounds that I need to drop to be back in the normal weight category (right now, I’m still overweight for a 6’1″ dude).  So, this year I resolve to actually start that exercise at least a couple of times a week to move that body-fat percentage down even more, and hopefully lose those last 10 lbs in the process.  I’ve been thinking that getting back into running again might do the trick.  However, Suzy and turned me on to this whole rebounding thing which involves a trampoline (how fun is that?).  While I feel pretty silly effectively bouncing on a tiny trampoline (and probably look pretty silly too), it leaves my muscles sore and jelly-like after only 20 minutes of jumping.  Seriously… even my abdominal muscles are sore.  NASA can’t be wrong about this… 

For additional motivation, I’m posting my 2011 data here (call it a public humiliation diet tactic).  I apparently love fads… slow-carb, rebounding, public humiliation.  I’m such a chump.

 

challenge accepted

Uncle Rick threw down the gauntlet during our yearly Parmeter Thanksgiving get-together, deep in the forested hills of Oregon. A challenge to post something (even if it is only a weak one liner) to their respective blogs. Well, here you go. It’s not much, but it will do. Now Uncle Rick, let’s see if you can match your challenge as well.

remote web file manager for freenas

Another geeky freenas related post today.  All those not interested in reading through techy garbage, please feel free to leave early.  For those of you here to learn about said techy garbage, please open your text books to page 204 and follow along.

For some time now, I have been aware of the rather handy feature that Windows Home Server boasts with a web based file manager (I believe they call it Remote Web Access or some such generic name).  Being a stalwart FreeNAS guy, I was impressed, but didn’t let my jealousy shine through.  How cool is it that you can log into your NAS from outside the network and upload/download any file that is there via a spiffy web interface?  Very cool.  Want.

Sure, you can setup an FTP server, but I really don’t like exposing an FTP to the outside world.  Too many security issues with FTP.  Yes,  I know you can setup an SSH tunnel and do FTP, but that’s kinda boring and not everyone has the patience to setup their own SSH client, etc. Plus I wanted an easy way to allow my friends and/or family to log into the server to grab things like photos or videos or even upload their own files.  I’m certainly not going to get 99% of these folks to setup a secure FTP session via SSH.

Recently, I came across this LifeHacker article about setting up FreeNAS to run SickBeard and SABnzbd to do a sort of Internet PVR.  After reading this, and seeing the techniques they used to get this up and running I was suddenly inspired to work on the above project.  I knew there had to be a way to shoehorn something in.  So I began to look at my options.  I finally stumbled upon a slick PHP/Ajax based Web File Manager front end with decent security that would work for me and decided to give it a shot.  (Ajaxplorer)

If you want to be the master of your domain, and show your Windows Home Server friends up, then this setup is for you.  Enough background, let’s get started.

Get the Necessary Pieces:

  • Step 1: Go to the Ajaxplorer website and download the latest zip (http://ajaxplorer.info).  As of the publication of this post, I downloaded Ajaxplorer v3.2.4 and I should also mention that I am using FreeNAS 0.7 in my configuration.  (sorry FreeNAS 8 folks, but I’m guessing these steps will work equally well on your server?)
  • Step 2: No step two?  Alright, on with it!

Configure FreeNAS:

  • Step 1: Open the admin web interface and navigate to the “Services” tab.
  • Step 2: Open “Webserver”
  • Step 3: Check the box to “Enable” your LightHTTPd webserver in the upper right hand corner (this doesn’t actually instantly enable anything, just allows you to start configuring)
  • Step 4: Choose HTTP from the protocol drop down (at least initially – we’ll talk about securing this before we poke a hole in your router and expose the NAS to the internet)
  • Step 5:  Select a port number (i.e. 1234).  (don’t select port 80, try not to conflict with other ports on the server and of course something between 0-64999)
  • Step 6: Choose webserver base directory (or document root).  I put mine on my mounted drives, rather than installing directly on the flash drive (OS drive).  For example:  \mnt\sharename\www\ (of course you will need to go and create this directory on your drive first)
  • Step 7: Leave Authentication and Directory Listing un-checked.
  • Step 8: Click the “Save and Restart” button.  You should get message stating: “The changes have been applied successfully” after a few tense seconds.

Install Ajaxplorer

  • Step 1: Unpack the ajaxplorer-3.x.x.zip into a directory off of your webserver base.  For example “. \www\ajxp”.  Confirm that all of the directories and files were created properly and that permissions were set properly (see this page for more details: http://www.ajaxplorer.info/wordpress/documentation-3/chapter-basics/ ).
  • Step 2: Point your browser to your FreeNAS IP and new webserver port (i.e. http://192.168.1.101:1234/ajxp) to get started.   A diagnostic tool will be ran once to check that your configuration is ok.  On FreeNAS you’ll get some warnings letting you know that you won’t have image previews due to the lack of PHP5, etc.  Despite these warnings you should have a mostly functional web file manager installed.
  • Step 3: Click continue and you should see a warning telling you that the user « admin » has been created with the password « admin » : log in with this id/password, and go the the « Settings » panel to change the admin password.
  • Step 4: Configure your users and repositories.  (http://www.ajaxplorer.info/wordpress/documentation-3/chapter-features/gui-overview/ )

Additional Security Concerns

I’m not much of a security expert, however it was absolutely essential to me that this be a decently secure solution.   I really don’t want people hacking around in my NAS from the outside.

In this configuration there are at least 3 caveats with regards to security:

  1. When using this interface over the internet, the traffic is unencrypted (as is your login and password).  You’ll want to setup a secure connection to your FreeNAS box and the easiest way to do this is to configure HTTPS on your box.  (http://www.freenaskb.info/kb/?View=entry&EntryID=284 ).  You’ll be self-signing your certificate, which means that your credentials will not be recognized by any reputable certificate authority – and that your browser will likely throw up all kinds of alarm bells when accessing your site.  But since you know you’re trustworthy, you can just ignore those warnings.  Do this step first such that you can generate the credentials, and the configuration file for your webserver – in the next step we’re going to use the newly created configuration as a base for a new webserver.  Just read on…
  2. FreeNAS’s LightHTPPd does not support .htaccess files (which are for Apache), which means that you’ll need to go and lock down any sensitive directories (i.e. .\server\conf, etc. ) manually.  Although it seems it may not be necessary after poking around a bit more, better safe than sorry.  More information can be found here: http://redmine.lighttpd.net/wiki/1/Docs:ModAccess.  I found that there are two configuration files under FreeNAS.  One for the WebUI and one for the web server that you just enabled.  You’ll want to edit the configuration file found here: \var\etc\websrv.conf.  Add something like this to the end of the file:

    $HTTP[“url”] =~ “^/ajxp/server/conf” {
    url.access-deny = (“”)
    }

    Now, to my surprise, if you disable and re-enable the webserver via the web UI or reboot the system you’ll completely blow away any changes you made to the config file (http://sourceforge.net/apps/phpbb/freenas/viewtopic.php?f=86&t=1941). This is apparently how FreeNAS is designed. So, here’s how I got around things. I made a copy of the websrv.conf file named mywebsrv.conf and instead of launching the default web server, launched my own via a config script. Do this by adding the following to System > Advanced > Command Scripts:

    /usr/local/sbin/lighttpd -D -f /var/etc/mywebsrv.conf -m /usr/local/lib/lighttpd

    Make this a post-init script meaning it runs after the system boots and initializes.  Ensure to add all of the directories that you would like to deny access to into the configuration file.  After you’re done with this step, make sure you disable the other default webserver via the web gui before you reboot the system. This should do the trick and forbid access to your sensitive directories.

  3. Ajaxplorer comes with a standard Admin account creatively named ‘admin’ so you’ll probably want to change that to avoid dictionary attacks.  I created a new account with admin privileges and a strong password and then deleted the admin account.

If you’re reading this and can think of more, please do let me know so that I can plug those holes too.

Poke a Hole in your Router

Because everyone has a different type of router at home, I won’t even attempt to begin to tell you how to do this.  However, you’ll want to make sure that you port-forward whatever port you defined in your LightHTTPd configuration above.  You can use a different outside port if you desire, but I like to keep it simple and use the same port number.

Now you can access your stuff anywhere you have an Internet connection.  If you have a Dynamic DNS service to help you remember your IP, even better!

Final Thoughts

While I’m pretty happy with the results, I’m sure there are improvements that can be made to the installation etc.  Others have used Ajaxplorer in place of the Quixplorer that is currently running on FreeNAS (which can be used on the internal network).  Ajaxplorer can be a bit sluggish, and it doesn’t integrate perfectly with FreeNAS, but it gets the job done.  If you have any suggestions, tweaks or improvements let me know and I’ll include them here!  I hope this helps anyone that has been thinking of doing this.  It’s certainly been a learning experience for me.

waffle

Remember a couple of years back where I made a rather bold, stake-in-the-ground, statement in a blog post?  Back then, I had such a different, pre-dad point of view.  It was such an innocent time.  A time with boundless amounts of energy, optimism and vanity.  At the time, it was all about maintaining that youthful, kid-less image.  It was about caring about style over function, performance over comfort.  At the time, a cross-over SUV seemed like a way to cling to those values, while getting some small level of practicality.

Mark your calendars (if you’re keeping track at home).  On August 12th, 2011 my will was finally broken and I have become a mini-van owner.  This post-dad point of view is 100% on-board with a mini-van and the lifestyle that it promises.  Mini-van’s are all about convenience and comfort.  They’re all about capacity and doors that open themselves with the push of a button.  They’re all about cup holders and arm rests and tinted windows.  They are about incrementally improving the family travel experience.

Just like last time, this is a new era.  An era that I fully admit that I resisted before but have come to completely embrace.  Next time I’m thinking about taking a stand on the topic, I’ll just remember this humbling moment when I waffled.

 

best of june 2011

Continuing with the monthly video theme, I managed to spend some time this weekend working on the video for June.  In this video, you’ll see a family camping trip to Grover Hot Springs (mainly the nearby falls), painting a clay pot in the backyard, the zoo and one quick photo of C and Suzy at a friends wedding.  A fairly short one this time, but action packed.

Again, if you’re interested in the technical side of things, I decided to give Kdenlive a spin this time, creating the entire clip with that software.  It is yet another outstanding video editor that has come to Linux recently – and I am thoroughly impressed with its capabilities.  It’s definitely got more of a learning curve than OpenShot, but it’s still reasonably intuitive.  I found that it is better at handling Pan & Zoom type effects (Ken Burns style) than OpenShot, with a much smoother result.  However, tiling photos and video is still something that I’ve found difficult to do – and has resulted in several crashes.  I eventually gave up on that effort.  The title generator is not as snazzy as OpenShot either.  But it seems to be snappier, and renders on the fly better than OpenShot.  I’m really torn between the two and may ping pong between them until I can decide which I like best.

July is going to be a much bigger video.  Until next time…

bittorrent on freenas

As previously mentioned, I am a big proponent of FreeNAS (a free network attached storage operating system). In the past I have talked about how to setup SubSonic (an awesome streaming audio/video server) on FreeNAS natively.

I am currently running the v0.7 (legacy) variant of FreeNAS (in my opinion v0.8 is still a bit young) which I recently “turned-on” the bittorrent capability.  This is a good thing, because, I can simply fire-up a torrent on the machine and let it download right onto my server (which is up 24 hours a day), instead of my desktop machine which may or may not be on all the time – and which may get updated with a new OS from time to time, blowing away all my seeding torrents.

While setting up bittorrent on FreeNAS is on the surface, really easy, I found some quirks that needed to be ironed out.  I decided to capture the information in one place in the hopes it can help someone else out there.  Here are the resources that I used to get this going:

1. Setting up the Transmission client on FreeNAS v0.7. A great tutorial can be found at LifeHacker, and even has a nifty video to help you set it up. (http://lifehacker.com/5822590/turn-an-old-computer-into-a-networked-backup-streaming-or-torrenting-machine-with-freenas).  Don’t forget to setup the DNS (this is critical for bittorrent to work at all).  Update (11/2011):  I found that permissions can be quirky with the standard setup – meaning if you’re trying to delete files via SMB/CIFS, you might get denied.  To avoid this issue, set the user mask to ‘0000’ in the Transmission setup page via the WebUI which will write your downloads as read/write/execute for all groups. 

2. Adding an IP block-list to Transmission under FreeNAS. Because I’m paranoid, I definitely want a block-list running on Transmission.  For some reason, this functionality is turned off by default in FreeNAS.  This helpful site, gives you the easiest method by which to turn it on and keep it updated.  (http://code.google.com/p/freenas-rss-extension/wiki/MyTweaksUpdate (11/2011):  You’ll want to update your blocklist URL to something more current as the bluetack URL does not seem to work anymore. (http://sourceforge.net/apps/phpbb/freenas/viewtopic.php?f=60&t=11473)

3. Using a remote GUI to control Transmission, monitor/start torrents. While some folks may find it perfectly acceptable to use the WebUI and a watch directory, as described in the LifeHacker article, I find that process kind of clunky.  I prefer to send the torrent files to a remote transmission client directly from my browser.  (for example, when I click on a torrent link, the browser downloads the torrent and launches the remote Transmission client automatically – no need to open up remote directories and pass files around).  Because I’m using Ubuntu on my home computer, I simply installed Transmission Remote GUI.  When clicking a torrent, simply point your browser at “/usr/bin/transgui” to start that software up.

That’s it folks. Have fun torrenting on your FreeNAS server!