This is a test of the WordPress.org site for blog posts.
This is a test of the WordPress.org site for blog posts.
Properties are an important part of determining and setting the configuration of ZFS storage systems. They can also be used to review the performance and usage of storage resources. Properties can be set at a top-level and inherited by child components when created. Understanding how ZFS properties are utilized is important to operating an efficient storage system.
In this article, we will explore how to properties work and how to get and set properties.
This article is one of a series of articles on ZFS. You can start at the beginning by creating a ZFS playground on which you can play.
In this talk, I lay out my vision for the use of unmanned and autonomous vehicles and their uses within city limits. There are many potential benefits in terms of public safety, disaster response, personal transportation, delivery and logistics. However, a balance is needed between safety and the ability of system developers to experiment and advance the technology in the environments in which these systems will be used. I explore many of these ideas in this TEDx Talk.
What do you think? In light of the proposed FAA regulations regarding unmanned aerial systems (released after my talk), did I present the issue with the right balance? Are unmanned and autonomous systems just too experimental at this point to be used in our cities?
I look forward to your comments.
In addition to data integrity, device redundancy, and performance features, ZFS Storage Pools can also expanded in usable storage size through deduplication and compression of the data stored. In other words by shrinking raw data and removing duplicated parts of data, ZFS Storage Pools can store more data on disk. While there are some memory trade-offs using deduplication, it can provide significant storage savings for some types of stored data. There are also some significant performance benefits to compression.
In this article, we will explore how to configure deduplication and compression for storage pools.
The OpenBSD‘s PF provides a great many features for packet filtering and network address translation. Mac OS X includes a version of PF that can be used to protect network services. In an article called “Mac OS X pf: Avoiding known bad guys“, I talk about using the Mac OS X PF firewall to protect against known bad sites. In this article, we explore a technique to protect SSH from attackers trying to gain remote access to your Mac by guessing passwords by brute force.
Most of what I learned about PF was used at the office to protect our network. We built redundant firewalls using FreeBSD and PF rules. I started to experiment with the Mac OS X PF implementation once I learned that it was shipped in Lion (Mac OS X 10.7). I am specifically concerned about my Mac laptop systems. The information in this article can also be applied to Mac server and desktop systems too.
ZFS provides several features for Storage Pools that can improve reliability of the storage system and increase the overall performance of reading and writing data. An additional level of reliability can be attained through the use of spare devices that can replace failed storage devices in a zpool. Data read performance can be increased through the use of cache devices. Data writes can be improved with the use of log devices. This article provides a description of each type of device and the commands through which they are configured.
Storage Pools are the basic method for consolidation of storage devices, data integrity, and redundancy for ZFS. Using some commands, you can quickly configure simple storage pools. In this article, we will explore the basic types of ZFS Storage Pools (stripes, mirrors, and RAID-Z), the available storage and redundancy trade-offs, and the commands used to create each type. There are several example commands included that can be used to create storage pools for experimentation and testing. Using the ZFS Playground we built in a previous article, we can experiment and test ZFS Storage Pools quickly and easily.
ZFS is one of those technologies that I have always had on my to-learn list. After I left Sun Microsystems in 2002, there was not much reason to go back to Solaris. I still had my Sun hardware for learning and experimentation. Soon after I started at Purdue University, I gave away and sold all of that gear to students that were interested in learning Sun hardware and Solaris.
When ZFS became available in OpenSolaris and Solaris 10 (update 6/06) in 2005 and 2006, respectively, I was no longer using Solaris or administering any Solaris systems. (If anything, I was complaining about the Solaris systems that I had to use.) As ZFS was integrated into FreeBSD, it became more interesting. I did not have any hardware that would be a good place to play and learn though.
Fast forward to the end of 2014, we now have access to a lot of retired server gear with plenty of disks, RAID cards, a lot of CPU and memory in the data center at work. So, I divided up the equipment with my security engineers as systems with which to learn and experiment. One of the first things I thought our group could use was a file server for desktop backups and general storage. Being a FreeBSD guy, my first thought was FreeNAS. After installing FreeNAS 9.3 and discovering that it was all ZFS now, I realized that now was the time to learn how to make ZFS go.
My initial research lead me to a great (but slightly old) presentation by Ben Rockwood called Becoming a ZFS Ninja (videos part 1 and 2). My eye-opening moment was when Ben talked about experimentation using VirtualBox and creating virtual disks to manage under ZFS. Well, of course. That seems obvious now!
I want to learn ZFS and play around in a safe environment. VirtualBox (or any other virtualization toolset) is a great way to do that. Sure, I have hardware now, but a virtualized environment to play with the configuration makes more sense. I expect to do stupid stuff in ZFS (some intentionally). What I need is an environment that is tolerant of mistakes and provides a easy way to go back and try something different. I cannot be driving over to the data center every time I mess up the base OS and have to reinstall again.
So, I built a safe place in which I can run around with scissors, bonk my head on the equipment, and jump off the swing set at the highest point. Nothing can hurt me because I can reset and try again. It is a safe playground for learning.
This post is really for me, but hopefully you can find something useful here as well.
We need to assemble our equipment and tools to build our playground. Here is a list of what my playground has. (You can build yours with similar equipment.)
That’s a very simple list. Feel free to deviate from it. For example, your playground may work equally well with other virtualization tools. I have access to VMWare Workstation at the office, but I never use it. VirtualBox is free to use and is actively supported. I chose FreeBSD 10.1 because it’s the new shiny FreeBSD release as of this writing. It also has ZFS baked into the OS. The BSD installer can also create a ZFS root partition, which I plan to experiment with as well.
You could use a different virtualization platform. In fact, I would interested in hearing about the use of other tools. Leave a comment.
You could use a different operating system. FreeBSD is something with which I am very familiar. There are ZFS implementations in several other operating systems. Pick your favorite. From what I can tell, most of the management of ZFS is handled using the
zpool commands, which are similar in most implementations.
The purpose of the playground is to learn the concepts in ZFS, play with the commands, and learn from mistakes. The playground is small though. We cannot build a large storage service with specific performance targets in the playground. Some features of ZFS cannot be enabled and used effectively given limitations of the playground (disks and memory mostly). We are also not attempting to build a file server with lots of file sharing services, like FreeNAS.
We can, however, build a reasonably good place to blow stuff up, get concussions, destroy data, and wreak havoc without anyone or anything being permanently harmed or any smoking hardware. We simply reset the VM back to a snapshot, and we are back in business.
The first thing that we need to do is to assemble our tools. If you don’t have your virtualization tools ready, go ahead and get those downloaded and installed. You can find VirtualBox at the virtualbox.org site, and it is available for Windows, Mac OS X, and Linux.
For the operating system, download the OS installer ISO image. For FreeBSD, download the amd64 (64-bit) ISO image (download). There are now many more options for installing 10.1, but this is all you need.
Configure your virtualization tool to create a new virtual machine (VM) for the ZFS-enabled operating system. Here are the VM specifications that I use:
While you are configuring the VM, you can also create a series of virtual disks for ZFS to manage. The flexibility of ZFS allows it to work with a variety of disks. For simplicity at this stage, I would recommend creating at least three virtual disks. Here is what I created:
The fixed size disks have their space allocated at creation, instead of growing the disk as it is used. The small size of the disks is not important and has no impact on ZFS for our purposes.
Assign the OS installation ISO image to the virtual CD drive and start the installation. For FreeBSD 10.1, you have the option to install the operating system on a root file system that is managed by ZFS. That’s great for future experimentation. To avoid confusion at this stage, let’s do the usual UFS installation. We may revisit FreeBSD on a ZFS root later.
The installation of FreeBSD 10.1 is left as an exercise for the reader. If you encounter issues though, let me know.
Once you have the OS installed, detach the installation ISO and reboot.
From here, I will be referring to FreeBSD 10.1 specifically.
The first thing we should do is check for updates and apply those. There may be updates that apply to ZFS, so let’s avoid potential issues by having the latest version available. Run the following command to download and install any FreeBSD updates.
FreeBSD out of the box has enough tools installed to make most admins happy. If you need more, fire up
pkg and install what you need. (This is also left as an exercise for the reader.) I would not waste too much time making this particular FreeBSD VM the ultimate admin world for you. We are here to learn ZFS, not FreeBSD specifically.
One of the first things you may notice is the FreeBSD kernel warning about ZFS not having enough memory to enable prefetch. That’s OK for now. It will still work for our purposes at this point.
Before we start playing, breaking, and doing relatively destructive things, let’s start with a VM snapshot.
Finally, we are at a point where we can start learning ZFS. Here is your first command:
Wait, what just happened? Well, that simple command created a new pool (“mypool”) containing our five virtual disks in a large stripe and mounted it.
No, really. Look:
Time to clean up. Use this command to delete the pool we just created:
Here’s what we did:
In a future post, we will use our new playground to run reckless and learn.
[Update January 5, 21:20: Josh Gillam find some grammar issues, which I corrected.]
TEDxLafayette 2014 was a wonderful experience for me personally. I was honored to have been asked by the committee to present.
I did not spend too much time on social media talking about the event ahead of time or even during the event. My focus was on putting together a great TEDx Talk and listening to the other great speakers.
I plan to post links to the video once it becomes available.
After the event was over, I found the following articles before and after the event. I also included links to some of the tweets during the event. If you find other sites and articles about the event, please send them to me.
On November 22, 2014 I participated in TEDxLafayette. Listed below is the prepared text that I worked from while preparing for this talk.
There was more content than I could fit into the time available (just eighteen minutes), so I had to cut back. The parts I left out are still listed below with strikethrough text.
When we talk about unmanned vehicles, most people think of drones. In particular, this kind…
Slide with Military Drone Firing Missile.
And it’s not surprising. Drones strikes are all over the news despite their mostly secretive use in combat zones.
Slide with Military Surveillance Drones
Military commanders are always interested in knowing more about the enemy. They want the ability to see over the next hill, to determine the composition, disposition, and strength of the enemy. The first uses of drones in modern warfare were focused on observation and surveillance. They are quicker to deploy and cheaper to operate than military spy satellites.
These small drones can be easily deployed in the field, literally by throwing them into the air. Once airborne, soldiers can remotely fly these drones over the target area to find and observe the enemy. With a remote control and a direct video feed from the drone, soldiers can locate enemy forces and determine their strength and readiness quickly.
Larger drones can be launched like traditional aircraft from standard air fields. They are designed to stay in the air for long periods of time to loiter over large areas. Drones such as the MQ-9 Reaper carry missiles and laser-guided bombs that can strike at the enemy. Their pilots can be located anywhere in the world.
Drone pilots operate in relative safety while flying. Drones that are shot at, shot down, or crash do not kill or injure the pilot. Pilots can even take breaks and hand the controls over to someone else during the flight. Larger drones that operate for long periods of time can rotate through several flight crews, each crew sleeping in their own bed at night.
Most of the younger soldiers are well-prepared for flying drones in the military having received their training from Xbox University and the School of PlayStation.
Slide with advanced drones.
Drones are also being used to solve problems in the changing battlefield. These drones are examples of the use of advanced technology to aid soldiers. The Black Hornet Nano on the left is designed to fly in tight quarters, look around corners, over walls, and around obstructions. Soldiers fly it with a remote control with an integrated video display. The T-Hawk drone on the right is in use by the British Army in counter-IED operations. The T-Hawk was also used at the Fukushima Dai-Ichi nuclear power station to survey the damage. The high radiation levels prevented humans from operating around portions of the plant, but this drone can go where people cannot.
Slide with the CBP drones.
The same basic military drones have been repurposed for use inside the United States.
NASA and NOAA have drones for research and environmental monitoring. In 2007, a NASA drone was used to survey Southern California wildfires and direct firefighters where they would be more effective. The Department of Homeland Security and Customs and Border Protection uses several drones like this one to patrol the Canadian and Mexican borders.
First slide of protests about killing children, legal questions, and STOP!!
The use of drones by the military has raised many ethical and legal questions. Many protest the targeted killing of people by remote control and the collateral damage and civilian casualties drones cause.
Second slide with a protest sign about drone terrorism and the use of drones on the border.
As those combat-focused drones are repurposed for use within our own borders, many more questions are raised about government intentions.
Last protest sign with Obama campaign look.
The appropriate use of drones on the battlefield in some distant land and here within our own country is set by policy at the very top levels of government.
There are many important issues and a wide-ranging debate is certainly needed. But that’s not what I am here to talk about.
I see a brighter future for unmanned vehicles in our society.
I see a more useful purpose in having unmanned and autonomous vehicles all around us.
One of the best uses of unmanned vehicles is to enhance public safety.
Slides with fires and fire truck.
Firefighters rely on their training and the experience of the senior firefighters and leaders. Upon arriving at the fire the team immediately begins to organize and plan how to attack the fire and save lives and property. If assistance is needed, a call is made for additional fire crews.
But what if these planning steps could occur before the firefighters arrived, maybe while in the truck on the way to the fire? Using an unmanned aerial vehicle launched from the fire station when the call comes in, the firefighters could receive detailed information about the size and scale of the fire before they arrive. Instructions and assignments could be given to the crew while in route. Additional equipment and crews could be called in if needed before the first truck pulls up.
Effectively, firefighters could have their own aerial surveillance plane flying over the fire providing real-time imagery. With temperature sensors, hotspots could be located. With chemical sensors, flammable and hazardous gases could be detected. People in need of rescue could be spotted.
Slides with police cars and SWAT team.
The entire police force in West Lafayette now use body-worn cameras. These types of cameras show a first-person perspective from the officer and provide the department and the citizens of the city with accountability and transparency. In addition to the police car dashboard camera systems, an officer’s direction interaction with the public can be recorded, situations can be deescalated, complaints against officers can be proved or disproved, and additional evidence for court is available.
Body-worn cameras, like those used by the West Lafayette Police Department, and dashboard cameras provide information after the fact though. Police officers may need information in advance of arriving at the scene of an emergency call. Situational awareness can be critical in saving lives. Unmanned aerial vehicles could provide situation awareness by being deployed when emergency calls are made, traveling quickly to the scene, and loitering over the area providing real-time video to dispatchers and officers. Knowing more about the situation before arriving can allow officers to be prepared and for backup to be called before the situation escalates.
Having an eye in the sky also allows suspects to be followed after fleeing a crime scene. If the robbers are speeding away or running through the alley before the police cars pull up, an unmanned aerial vehicle can be flying overhead following the suspects. A UAV is cheaper to operate than a helicopter, is smaller and quieter, and can provide real-time location data on the suspects. Smaller UAVs can be used inside buildings to assess hostage and standoff situations.
Unmanned aerial vehicles can also provide crime scene support and assist in searching for fugitives and missing persons.
Slide with traffic and car accident.
Most major roadways have sensors to detect the flow of traffic. Transportation officials use this information to monitor traffic volume.
You can even see this information in navigation apps that show you the flow of traffic. When you see a lot of red ahead on your map, you are probably going to be moving slowly for a while. Some apps even provide visual indicators of traffic accidents. While there are some traffic cameras at key intersections, there is no way currently to provide complete camera coverage for the entire roadway.
Take accidents for example. Most are reported through 911 calls. A good number of these accidents do not occur on camera. Emergency dispatchers rely on the caller to provide the details. The use of unmanned and even autonomous aerial systems could provide visual information quickly to dispatchers and emergency personnel.
The number and types of injuries could be determined before the emergency personnel arrive. Plans to attack a vehicle fire and rescue people could be made before firefighters jump out of their truck.
Slides with police command center, Red Cross disaster relief, and NRC command center
In emergency response and disaster relief situations the most valuable commodity is information. Natural disasters often destroy much of the communication infrastructure making it difficult to gather and share information about relief needs and directing the response. Sadly, we are all too familiar with natural disasters and emergencies in our own community.
Unmanned and autonomous vehicles could be deployed quickly in a disaster area to bring more information to the incident command centers so that decisions can be made quickly to save lives, protect property, and help a community heal faster. Using current technology, UAVs can provide high-resolution mapping of an area, just by drawing on a square on a digital map. UAVs could be used to loiter over a disaster area to watch for fires, use sensors to find missing persons, and to provide communications support to personnel on the ground.
While the military-industrial complex may have brought about the first drones, I think the way to bring these technologies to cities is through experimentation, innovation, and entrepreneurship. The wild-eyed dreamers and tinkerers in their garages, maker spaces, and university labs are creating cool technologies now that will fuel this vision in the future.
Slide with the hobbyist-built rovers and quadcopters.
This is the Maker Movement. Their tools are cheap electronics, kits, digital design tools, 3D printers, laser cutters, source code, and a pile of junk to source as parts.
Slide with the 3DR Iris+ and the hexacopter.
We have already seen what a hobbyist can do with an idea. These UAVs are bare-bones flying platforms with software. An app on your mobile device can provide the mission data to the UAV for autonomous flight. You can add your own sensors and payloads to make it do something useful.
A graduate student in Europe developed an ambulance drone for his masters research thesis. The first prototype carries an automated external defribrillator unit that can arrive within a few minutes of the emergency call.
Slide with the DJI Phantom.
This is a growing industry in unmanned aerial vehicles. We have quickly moved from a hobbyist with a kit to fully assembled and tested products that you can buy right now. The transition from first tinkering with home-built UAVs to today has been about seven years.
While there are still kits available, companies such as DJI and 3D Robotics are focusing on aerial photography and mapping solutions.
In the U.S. in 2012 more than 33,000 people died in vehicle accidents. While that number has been steadily declining, that’s still a large number of people. Vehicle safety systems such as airbag, crash avoidance, and driver assistance systems have helped lower that number. However, most accidents are related to driver error.
If we can eliminate the driver, can we prevent even more deaths?
For those that cannot drive, can we improve their mobility?
Slide with the Google Car with Eric, Sergey, and Larry
Most likely you have heard of the Google Self-driving Car. For many years research has been conducted to bring the driverless vehicle to reality.
And Google is not the only organization interested in this. Most major car manufacturers have announced strategies related to driverless cars as future products.
Slide with other Google Self-Driving Cars
There are still many challenges in bringing driverless cars to our cities. Bad weather, poor visibility, parking lots and garages, potholes, construction zones, pedestrians, bicycles, and reckless drivers are challenging enough for human drivers. For the current fleet of driverless cars, those are still issues to be solved.
Google is in a unique position in tackling these challenges. Consider that Google products such as Google Maps and its navigation service and Street View provide input to the self-driving car system. When new stop signs are added to an intersection, the car cannot only detect the sign and stop but also submit it as a new update to the map. These cars learn more about the environment as they drive through it.
Slide with Amazon and DHL drones.
Amazon, Google, DHL, and others are currently experimenting with package delivery direct to your home or business using UAVs. Delivery by drone could relieve some road traffic congestion and reduce greenhouse emissions as well as shorten delivery time.
As Amazon continues to build new distribution centers closer to the populations it serves, it can use those centers as hubs for UAV deliveries. Instead of a truck, a UAV would be loaded at the center with the products and a flight plan for the delivery. The UAV could deliver the package directly the customer’s doorstep and then return to the distribution center for a fresh charge and another delivery.
DHL is currently using its Parcelcopter in experimental deliveries to a remote island in the North Sea. It’s payload is medicine and travels when other forms of transportation are not available. DHL will be able to quickly improve the system over time as the North Sea is challenging location to fly because of the weather.
If you combine driverless vehicles and the need to deliver large volumes of goods, then you must also consider the possibility of autonomous long-haul trucks as well.
Slide with Chicago Sky line.
As unmanned and autonomous vehicles become more a mature and reliable technology, how do we prepare our cities for them?
Slide with the FAA logo.
For aerial vehicles, the Federal Aviation Administration is the protector of the national air space in the United States. Safety is the primary mission of the FAA. There are many concerns about unmanned aerial vehicles intermingling with aircraft in the sky.
The FAA defines three types of unmanned aerial system operations. A civil UAV use allows for research and development, but not for carrying people or property for compensation or hire. A Public UAV operation includes firefighting, law enforcement, disaster recovery, and search and rescue operations. For civil and public uses of UAVs there is an application, review, and approval process.
Model Aircraft is the designation for hobbyists and needs no special filing with the FAA.The current rules for hobbyist UAVs is that they have to operate below 400 feet, within sight of the operator, and away from airports. This affords the hobbyist an opportunity to experiment but limits the ability of innovators to turn their ideas into commercial operations.
Under these rules, it is not possible for anyone to provide unmanned aerial services for compensation. A company cannot legally sell you a video of your property taken from a UAV, deliver products to you, or even provide an aerial survey of your property or farmland.
In 2012, Congress directed the FAA to integrate unmanned aircraft systems into the national air space. The FAA has issued interim rules and allowed some commercial uses of UAVs. A final rule is expected later but the hope is that there will be a balance between safety and the entry of commercial UAV operations into the national air space.
Slide with the states allowing driverless cars.
This map shows the states that currently allow driverless vehicles on their roads. California, Nevada, Michigan, Florida, and the District of Columbia. There is still a lot of open space on that map.
The city of Coeur d’Alene, Idaho has a city ordinance allowing robotics and self-driving cars within city limits. The hope is that this will encourage robotics research and the city will become a hub for the robotics industry.
Slide with various obstructions (power lines, traffic lights, construction detours, street signs, etc)
But cities are messy places that make the safe operation of unmanned vehicles challenging. There are obstructions in the air and on the ground, changes in the environment that an autonomous vehicle must navigate, and of course the people that live there.
The current experimental self-driving cars use sensors to detect obstructions and watch for traffic signals. Aerial vehicles do not have similar sensors yet. First they aren’t allowed to operate in populated areas. Second, the weight of additional sensors and the computation required places limits on their ability to fly. However, UAVs operate with a flight plan. Flight planning software can create that plan and could incorporate avoidance of known obstructions automatically. As the weight of sensors decrease, aerial vehicles will be able to sense their environment and avoid obstructions automatically. Just as Google Street View cars drove through our cities capturing images from the street, similar mapping vehicles could be used to collect information about the city environment to aid in vehicle navigation and avoiding known obstructions. That database could shared with vehicle operators to aid in navigation and flight planning.
Slide with Lafayette Skyline.
State laws and city ordinances signal the start of an effort to integrate these technologies into our everyday lives. However, there is also a recognition that most of these technologies are still under development. They are not mature yet. It may take many years to get these technologies to a point of reliability and safety with which most people can be comfortable.
Just as early cars and airplanes had to develop and evolve over many years, we are only at the beginning stages of the implementation and distribution of this technology.
I propose that we need to have a strategy at the federal, state, and local levels for unmanned and autonomous vehicles. This strategy should consider all kinds of unmanned and autonomous vehicles and the environments in which they will operate.
First, we need to have public debates and discussions on how these technologies can benefit society and address some longstanding problems. Safety will always be a prime concern.
Privacy is already a concern for citizens and a reasonable use for the information collected by unmanned vehicles must be considered. We must balance this concern with the benefits of innovation to move us forward.
Second, we need to embrace experimentation now rather than wait for the technology to mature first. The reason is simple. Developers need to operate in real environments in order to solve problems. It is not possible to simulate all situations in a laboratory setting or even on the sunny, well-mapped streets of California.
Third, we need to create a legal framework that allows for the safe experimentation and operation of unmanned vehicles in populated areas. Our current frameworks define or assume that an operator is always in control of a vehicle. That may not always be case. We also need to provide a reasonable foundation in which new business models can develop. It is not always possible to predict how technology will be used nor should we try.
My hope is that we can find a way to bring the benefits of unmanned vehicles to our city. And soon.