Project Pat

Before anyone reads on - it looks daunting to me and I wrote it!

So have a look at this which is a great and more fun summary.


Apologies didn't see this message come through sir. Massively off topic now, but bear with me whilst I explain...

USB C is not what I need here, I need to cover off some other requirements, which will help explain my rationale. If I go the USB C route I will be limited to single drives and if I use large density cheap-ish spinning platters then I'll simply not get the speed out of them to perform editing. I would also have to be within arms reach of the drives - which will mess with my Zen like office (not really) but I don't want devices all over the place. If I use SSD's for the speed then it will cost a small fortune to get the amount of data storage I need. I've also invested in NAS since 2011 so it would seem silly to lose all the resiliency and have to use stand alone drives, bit of a retrograde step. The 250Gb's were a trial and were based on Sata so I didn't want to go all out, I've now switched to 2 x 500Gb NVME which is much better (speeds of between 1800-2700MB/s vs around 550MB/s on Sata), and as you say definitely the sweet spot at 500Gb

With 10GbE, or 5GbE as I'm sitting on now (more on that below), then I can have the storage up to 100m away from the computer as I'm running Cat7 cabling, and the data is accessible from anywhere I want it to be, or any device for that matter, even simultaneously.

I will be trying out multiple raid setups to get this running the best for me. Raid 1 over Raid 0 for speed and basic resilience, then Raid 5, 6 and 10 etc. All with and without caching and over 5GbE and 10GbE.

Even on basic Raid 5 with no NVME acceleration (and now over 5GbE - as I have moved onto a larger NAS with a better blend of capabilities but lower interface speed) I'm getting the speeds below - which is just about enough to get me to 4K editing. The 5GbE interface will tap out at roughly 500MB/s which is more than ample for editing anything in the 4k range that I can produce.

I also have the option on the switch to run LAPC port aggregation which I have tried on my old NAS with 4 x 1GbE and that worked ok, but it was time for a performance upgrade so I hit the reset button and went for some new kit.

To give an idea of the file sizes I'm looking at. I have imported the video that I took originally for the tow bar work (I'll do the work over again to get the detail right - I'm just learning - but the data size is useful if nothing else). Once imported and made into proxy media we are looking at the thick end of 500Gb of data. I can remove the proxy media when I archive it but that's a significant amount for 1 video and not all of that was at 4k either. I have been known to over-egg the pudding so to speak but I prefer to buy the right things and use them for a long time than to cut corners at the start and either not achieve my goals or be delayed re-visiting things every 5 minutes. Wether I decide to do lots of video's here (I started last year on basic door removal etc but not yet published) or just go for home video editing then

Here is a link to the new NAS I'm running from QNAP. So far a great device - https://amzn.to/338x7qu

Love the feedback and happy to take offline or to the green room as waaaaaaaay off topic now.

View attachment 57234

You are not limited to single drives with usb c you can use a range of enclosures don’t use raid one if you are performance focus as you will get a little less than the performance of a single drive you want to us raid 10, don’t use raid 5 it always ends in tears, Raid 6 is ok but for speed raid 10 really is the only option if you want redundancy.

Simply the cost of the 10Gbit networking is a big factor. If you can avoid it I would.

I work in data centre computing as well as consumer computing for businesses.

As you have discovered your NVME isn’t limited by SATA bus however with a 6 or 8 disk array and the cost of the SATA drives you will max out your connectivity before the storage.

The other option is to use a clever SAN that uses platters and SSDs that will give you a lot of density without much of a hit in performance.

The Intel 660p NVME are interesting drives coming in at 1Tb for £100 but it cheats by using slow solid state with fast solid state so for your usage speed will drop off during sustained copying.

The QNAP san and nas are good. The other option. Is to keep your storage and processing local. And access the processing remotely.




Sent from my iPhone using Tapatalk
 
I hear ya. I would say I'm not unfamiliar with IT myself ;) possibly for the same reason as yourself.

I've been on Raid 5 plus cloud backup for 6 years without any hitches whatsoever (and I have had failures) so I guess my mileage varies wrt to Raid 5 being dreadful. I too have heard a lot of horror stories but these have mostly been due to the lack of a sound backup strategy (which Raid is not).

For me raid needs to put out upwards of 250MB/s to meet my needs and I'm almost there with Raid 5 so I'm pretty certain that a few more drives and some NVME Raid 0 caching and I'll easily get both that and the IOPS required.

Back to the raid for a sec. Raid 1 for resilience over Raid 0 for speed is what I was getting at. 2 x disks in Raid 0 giving the combined read speed and slightly less write with another 2 in Raid 0 with Raid 1 over the top for resilience. Works fine, but you lose 50% space, which is not efficient.

Access over the network really is a killer feature, I don't want to be labour the point but being tied to one machine is no fun and the storage has multi use for the family over the network that way too (although I may forget to tell them it's there!). 10GbE is not really as bad as you think cost wise, I spent just as much on Coax in 1991 easily and that ran at 10Mbit. It was £90 to upgrade my computer to 10GbE and the switch was £200 so not earth shattering by any means. 10GbE is really becoming mainstream in NAS this year and I expect that even the cheaper multi drive devices will sport >1GbE next year. I also expect the switches to come down in price in the next 12 months (they have already) with many new devices coming to market the price will naturally fall.

The NVME really is just to play about, the new NAS' feature QTier which can move data around depending on access needs, so that it can go from slow HDD on Raid, through caching Sata SSD's through to super fast NVME SSD's. I'm going to give it a try and see how it works, but for working on larger projects 1Tb of NVME Raid0 cache should make a massive difference. As you point out this will easily saturate 10GbE but given the investment it will make the most of the connectivity until 25GbE / 40GbE comes along - which is on the upgrade path for the NAS' also.

I looked at the Intel devices and as you say they are great in theory but if you hit them hard then they quickly run out of performance, the NVME I chose at are the cheaper end of the scale so not blazing but easily able to keep up with the 10GbE connectivity. Although given the speeds I'm seeing at present I'll be interested to see what 'actual' difference they make in real world computing.

Local storage is not an option for me, the device I'm working on simply doesn't have the space to accommodate large storage, although it is all NVME based and therefore blazing fast it is also frightening expensive (I'm sure from this you can work out what I'm running on)

Here are my local speeds.

Local speed.jpg


All in all I'm quite happy as this upgrade was cost neutral for me (or very near). I had lots of older equipment and it simply wasn't cutting it for me, so I've streamlined the data centre (loft) and also my lab (bedroom) and the difference is huge.

I now have no excuse not to get on with making some video's, the only problem is that I have no clue about film making. It's a steep learning curve for sure. I'm really really grateful for your advice, it really is great to hear a different perspective, there is no perfect answer, just a list of objectives and compromises. :)
 
I hear ya. I would say I'm not unfamiliar with IT myself ;) possibly for the same reason as yourself.

I've been on Raid 5 plus cloud backup for 6 years without any hitches whatsoever (and I have had failures) so I guess my mileage varies wrt to Raid 5 being dreadful. I too have heard a lot of horror stories but these have mostly been due to the lack of a sound backup strategy (which Raid is not).

For me raid needs to put out upwards of 250MB/s to meet my needs and I'm almost there with Raid 5 so I'm pretty certain that a few more drives and some NVME Raid 0 caching and I'll easily get both that and the IOPS required.

Back to the raid for a sec. Raid 1 for resilience over Raid 0 for speed is what I was getting at. 2 x disks in Raid 0 giving the combined read speed and slightly less write with another 2 in Raid 0 with Raid 1 over the top for resilience. Works fine, but you lose 50% space, which is not efficient.

Access over the network really is a killer feature, I don't want to be labour the point but being tied to one machine is no fun and the storage has multi use for the family over the network that way too (although I may forget to tell them it's there!). 10GbE is not really as bad as you think cost wise, I spent just as much on Coax in 1991 easily and that ran at 10Mbit. It was £90 to upgrade my computer to 10GbE and the switch was £200 so not earth shattering by any means. 10GbE is really becoming mainstream in NAS this year and I expect that even the cheaper multi drive devices will sport >1GbE next year. I also expect the switches to come down in price in the next 12 months (they have already) with many new devices coming to market the price will naturally fall.

The NVME really is just to play about, the new NAS' feature QTier which can move data around depending on access needs, so that it can go from slow HDD on Raid, through caching Sata SSD's through to super fast NVME SSD's. I'm going to give it a try and see how it works, but for working on larger projects 1Tb of NVME Raid0 cache should make a massive difference. As you point out this will easily saturate 10GbE but given the investment it will make the most of the connectivity until 25GbE / 40GbE comes along - which is on the upgrade path for the NAS' also.

I looked at the Intel devices and as you say they are great in theory but if you hit them hard then they quickly run out of performance, the NVME I chose at are the cheaper end of the scale so not blazing but easily able to keep up with the 10GbE connectivity. Although given the speeds I'm seeing at present I'll be interested to see what 'actual' difference they make in real world computing.

Local storage is not an option for me, the device I'm working on simply doesn't have the space to accommodate large storage, although it is all NVME based and therefore blazing fast it is also frightening expensive (I'm sure from this you can work out what I'm running on)

Here are my local speeds.

View attachment 57241

All in all I'm quite happy as this upgrade was cost neutral for me (or very near). I had lots of older equipment and it simply wasn't cutting it for me, so I've streamlined the data centre (loft) and also my lab (bedroom) and the difference is huge.

I now have no excuse not to get on with making some video's, the only problem is that I have no clue about film making. It's a steep learning curve for sure. I'm really really grateful for your advice, it really is great to hear a different perspective, there is no perfect answer, just a list of objectives and compromises. :)

What I mean by moving the storage to the processing is that you work on the machine remotely for tasks that require high iops such as using Citrix or rdp over a lan you don’t see much loss. That way you don’t have to be tied to a high powered machine and you can access from anywhere an use cheap hardware on the workstation side


Sent from my iPhone using Tapatalk
 
Ahh I see, that would work but I'm thinking heavy video work on RDP with colour grading doesn't work for me personally. ?
 
Ahh I see, that would work but I'm thinking heavy video work on RDP with colour grading doesn't work for me personally.

Disney use Vmware horizon Remote Desktop interesting enough so they can centralise their compute


Sent from my iPhone using Tapatalk
 
I'm on fire now! That is the roof done, just one more side and the rear and it will be ready for the oily stuff being tackled.

The footage in this makes me laugh, it looks like the car is breathing at some points!!!

Little quiz, can anyone figure out what I have done to cause this? (ps. it won't happen again!!)

 
I had to stop watching, as I was starting to feel seasick.

The answer is probably some sort of motion sensing software built into the camera that you are using (ideal for example when you're riding on a bike and want it to feel smooth), but in this case it just makes the whole thing feel o_O?

I'd suggest finding the setting and TURNING IT OFF :)

Otherwise, when can you come down and do Baby Boomer ;)
 
I had to stop watching, as I was starting to feel seasick.

The answer is probably some sort of motion sensing software built into the camera that you are using (ideal for example when you're riding on a bike and want it to feel smooth), but in this case it just makes the whole thing feel o_O?

I'd suggest finding the setting and TURNING IT OFF :)

Otherwise, when can you come down and do Baby Boomer ;)

It is even more bizarre than that!!! I used my drone in the garage to get a much better (apparently!) shot of the car as it was being cleaned. I then used Final Cut Pro to "stabilise" the footage and it ended up like this!! I may have to re-do this video with the footage stabilised properly, but to me it looked like the car was breathing at some points!!

The take away here is "don't be smart ar$£" and just film it on a tripod! ;)
 
And any time I can be of service cleaning cars Jeremy I'm more than happy to help, although from what I see you definitely don't need any :)
 
I used my drone in the garage

Ah, that explains why the camera keeps moving (vertically and horizontally) relative to the car! I had started to think of some sort of motorised mount and auto-tracking software but it still didn't explain everything. In the end the mount was motorised ... just not attached to anything! :D
 
Ah, that explains why the camera keeps moving (vertically and horizontally) relative to the car! I had started to think of some sort of motorised mount and auto-tracking software but it still didn't explain everything. In the end the mount was motorised ... just not attached to anything! :D

I should have taken a video of the drone taking the video :)
 
Back
Top