CLI File Editing with NetApp Data ONTAP 7-Mode

Let me preface this by saying that it’s really not an expectation that you should, or even that you’re going to be, reading or modifying files from the storage array’s CLI. Files are meant to be accessed using a (surprise, surprise) file access protocol, such as NFS or SMB/CIFS, directly from a client. Having said that, it can be useful to use the CLI to modify system files such as /etc/rc or /etc/hosts, particularly during initial setup or maintenance activities, and I’ve done it myself hundreds if not thousands of times.

The biggest hurdle to overcome in CLI file editing is that there isn’t a real editor included in the shell. Nope – no vi or Vim, no emacs, nano, or any other editor you may expect or like. There isn’t even ed! Instead, file access and editing is done using rdfile to “read” or display the contents of a file, and wrfile to “write” the contents of a file. Rdfile is pretty straightforward – it’s essentially just a cat equivalent that takes a file and sends it to standard output (your screen).

mba-7m-1> rdfile /etc/hosts
#Auto-generated by setup Mon Nov 3 21:54:55 GMT 2014
127.0.0.1 localhost localhost-stack
127.0.10.1 localhost-10 localhost-bsd
127.0.20.1 localhost-20 localhost-sk
172.16.213.18 mba-7m-1 mba-7m-1-e0a

And no, before you ask: there is no included equivalent of more or less (for paging of output), nor anything like tail or head. More’s the pity.

Wrfile, on the other hand, is a little different and less friendly. It does the exact opposite of rdfile – it takes standard input (the keystrokes/characters you type into the console/shell) and sends (writes) it to a file. This makes it very easy to create a file – as long as you don’t make any typos! There is no interactive line editing with wrfile – if you make a mistake you’re essentially starting over. Your best course of action is to create your text locally on your workstation with your tool (Vim, notepad++, etc.) of choice, and then paste it into your terminal window.

After you’ve finished entering the desired content, make sure to hit Enter to be on a new line and then simply use Ctrl+c to end the input and terminate wrfile. If you aren’t on a new line, any text already entered on that line will not be saved. Don’t mind the “error reading standard input” – that’s normal.

mba-7m-1> wrfile /etc/thisisonlyatest
blah blah blah
line 2 blah blah blah
read: error reading standard input: Interrupted system call
mba-7m-1> rdfile /etc/thisisonlyatest
blah blah blah
line 2 blah blah blah

You need to be especially careful using wrfile to edit an existing file: it will overwrite the contents of the file you specify as soon as you hit Enter. Even if you don’t type anything and immediately Ctrl+c out, the damage is done and your file is now empty. For this reason, always be sure to rdfile the file prior to making any modifications to it so that you have a copy in your cache. In most cases, you’ll want to copy the contents printed out via rdfile to a local texteditor for modification; you’ll then copy the modified content and paste into the console window after starting the wrfile operation.

mba-7m-1> wrfile /etc/thisisonlyatest
read: error reading standard input: Interrupted system call
mba-7m-1> rdfile /etc/thisisonlyatest
mba-7m-1>

If all you need to do is to add a line (or lines) to a file, it’s best to use the “-a” parameter for wrfile in order to append to the existing file rather than overwriting it. This also works for creating a new file.

mba-7m-1> rdfile /etc/thisisonlyatest
/etc/thisisonlyatest: No such file or directory
mba-7m-1> wrfile -a /etc/thisisonlyatest Line1 foo
mba-7m-1> wrfile -a /etc/thisisonlyatest Line2 bar
mba-7m-1> rdfile /etc/thisisonlyatest
Line1 foo
Line2 bar

File editing via the Data ONTAP CLI is not a normal operation, it’s not very elegant (ok, it’s crude), and it’s really only intended for atypical administrative use-cases – but it gets the job done when you need it. Otherwise, follow the recommended method for modifying files by using a licensed file access protocol (NFS or SMB/CIFS) and edit from a remote client instead.

Tours of the Black Prompt: NetApp Data ONTAP 7-Mode

NetApp’s FAS series of storage arrays run a single operating system called Data ONTAP across all platforms – from the low end to the high end – though this operating system can run in one of two different modes. There’s the older “classic” mode, which the majority of FAS administrators are used to and which is now referred to as Data ONTAP 7-Mode, and there’s the new, scale-out mode that is simply called clustered Data ONTAP. While the two modes share similar administrative concepts and some similar commands, the two CLI operating environments are quite different in behaviors, structure, and feature sets. We’ll start by talking about 7-Mode.

Commands and Privilege Levels

Logging in (via SSH, telnet, or console) you are provided a simple administrative-level prompt:

 mba-7m-1>

The prompt contains the hostname of the controller (“mba-7m-1”) followed by a “>” which indicates that you are in the normal administrative privilege level.
From here, you can see the available commands by simply typing “?”:

?                   flexcache           options             smtape             
acpadmin            fpolicy             orouted             snap               
aggr                fsecurity           partner             snaplock           
arp                 ftp                 passwd              snapmirror         
autosupport         halt                ping                snapvault          
backup              help                ping6               snmp               
bmc                 hostname            pktt                software           
cdpd                httpstat            portset             source             
cf                  ic                  priority            sp                 
charmap             ifconfig            priv                stats              
cifs                ifgrp               qtree               storage            
clone               ifstat              quota               sysconfig          
cna_flash           igroup              radius              sysstat            
config              ipspace             rdate               system             
coredump            iscsi               rdfile              timezone           
date                key_manager         reallocate          traceroute         
dcb                 keymgr              reboot              traceroute6        
df                  license             restore             ucadmin            
disk                lock                restore_backup      ups                
disk_fw_update      logger              revert_to           uptime             
dns                 logout              rlm                 useradmin          
download            lun                 route               version            
du                  man                 routed              vfiler             
dump                maxfiles            rshstat             vlan               
echo                mt                  sasadmin            vmservices         
ems                 nbtstat             sasstat             vol                
environment         ndmpcopy            savecore            vscan              
exportfs            ndmpd               sectrace            wcc                
fcadmin             ndp                 secureadmin         wrfile             
fcnic               netdiag             setup               ypcat              
fcp                 netstat             sftp                ypgroup            
fcstat              nfs                 shelfchk            ypmatch            
fcvi                nfsstat             sis                 ypwhich            
file                nis                

This shows you all of the commands available for this administrator privilege level.

There are two additional privilege levels available: advanced and diag. You can change to these levels using the “priv set” command.

 mba-7m-1> priv set advanced
 Warning: These advanced commands are potentially dangerous; use
 them only when directed to do so by NetApp
 personnel.
 mba-7m-1\*>
 mba-7m-1\*> priv set diag
 Warning: These diagnostic commands are for use by NetApp
 personnel only.
 mba-7m-1\*>

The asterisk after the hostname indicates that you are in either advanced or diag privilege mode. There’s no visual distinction between the two modes, but you can find out which one you are in by typing simply “priv”:

 mba-7m-1\*> priv
 diag
 mba-7m-1\*>

More commands are available within the higher privilege levels than in the normal admin level.

Advanced:
/etc/rmt            ftp                 nfsstat             sldiag             
?                   getXXbyYY           nis                 sm_mon             
acorn               halt                nv8                 sm_mon_old         
acpadmin            hammer              ontapi              sm_not             
aggr                help                options             smb_hist           
arp                 hostname            orouted             smtape             
autosupport         httpstat            panic               snap               
availtime           ic                  partner             snaplock           
backup              if_addr_filter_info passwd              snapmirror         
blink_off           ifconfig            perf                snapvault          
blink_on            ifgrp               ping                snmp               
bmc                 ifinfo              ping6               software           
bootfs              ifstat              pktt                source             
bringhome           ifstat_test         portset             sp                 
cdpd                ifswitch            priority            statit             
cf                  igroup              priv                stats              
charmap             inodepath           ps                  storage            
cifs                ipspace             qtree               stsb               
clone               iscsi               quota               stty               
cna_flash           key_manager         radius              sum32              
cna_flash_image_reset keymgr              rdate               sysconfig          
cna_flash_image_set l2ping              rdfile              syslog             
cna_flash_version   led_off             reallocate          sysstat            
com                 led_off_all         reboot              system             
config              led_on              registry            systemshell        
coredump            led_on_all          restore             tape_qual          
cxgbtool            led_on_off          restore_backup      test_lcd           
date                led_reset_all       result              timezone           
dcb                 led_test            revert_to           toe                
dd                  led_test_one        rlm                 traceroute         
df                  license             rm                  traceroute6        
disk                lock                rmt                 ucadmin            
disk_fw_update      log                 route               ups                
disk_list           logger              routed              uptime             
disk_stat           logout              rpc_stats           useradmin          
dns                 ls                  rshkill             version            
download            lun                 rshstat             vfiler             
du                  mailbox             rsm                 vif                
dump                man                 rtag                vlan               
echo                maxfiles            rtfo                vmservices         
ems                 mbstat              sasadmin            vol                
environ             mem_scrub_stats     sasstat             vscan              
environment         memerr              savecore            vstorage           
exit                mt                  scsi                waffinity_stats    
exportfs            mv                  sectrace            wafl               
fcadmin             nbtstat             secureadmin         wafl_backdoor_stats
fcnic               ndmpcopy            setup               wafl_susp          
fcp                 ndmpd               sftp                wcc                
fcstat              ndp                 sh                  wrfile             
fcvi                netdiag             shelfchk            ypcat              
file                netstat             showfh              ypgroup            
flexcache           nfs                 showfh4             ypmatch            
fpolicy             nfs_hist            sis                 ypwhich            
fsecurity
Diag:
/etc/rmt            fcstat              netstat             sm_not             
?                   fcvi                nfs                 smb_hist           
acorn               file                nfs_hist            smf                
acpadmin            filersio            nfsstat             smtape             
aggr                flexcache           nis                 snap               
anvl                fpolicy             nm                  snaplock           
arp                 fru_led             nv                  snapmirror         
autosupport         fsecurity           nv8                 snapvault          
availtime           ftp                 ontapi              snmp               
backup              gdb                 options             software           
blink_off           getXXbyYY           orouted             source             
blink_on            halt                panic               sp                 
bmc                 hammer              parityck            spares_zero        
bootargs            help                partner             spinhi_stats       
bootfs              hostname            passwd              spinnp_replay      
bringhome           httpstat            perf                spinnp_replay_stats
cdpd                ic                  ping                stack              
cf                  icbulk              ping6               statit             
ch                  if_addr_filter_info pktt                stats              
charmap             ifconfig            portset             storage            
cifs                ifconfig_priv       printflag           stsb               
clone               ifgrp               priority            stty               
cna                 ifinfo              priv                sum32              
cna_flash           ifstat              prof                sync               
cna_flash_image_reset ifstat_test         ps                  sysconfig          
cna_flash_image_set ifswitch            qtree               syslog             
cna_flash_version   igroup              quota               sysstat            
com                 inodepath           radius              system             
config              iomem               raid_config         systemshell        
coredump            ipspace             rastrace            tape_qual          
ct_dump_t3          iscsi               rdate               tcp_client         
ct_phy_read         iswt                rdfile              tcp_server         
ct_phy_read_t3      key_manager         reallocate          test_lcd           
ct_phy_write        keymgr              reboot              time               
ct_phy_write_t3     kma_stats           registry            timezone           
ct_reg_read         kt                  restore             toe                
ct_reg_read_t3      l2ping              restore_backup      traceroute         
ct_reg_write        label               result              traceroute6        
ct_reg_write_t3     labelmaint          revert_to           treecompare        
ct_reset_t3         led_off             rlm                 ttcp               
ct_tpi_par          led_off_all         rm                  ucadmin            
ct_tpi_read         led_on              rmt                 udp_client         
ct_tpi_write        led_on_all          route               udp_server         
cxgbtool            led_on_off          route_priv          ups                
date                led_reset_all       routed              uptime             
dbg                 led_test            rpc_stats           useradmin          
dcb                 led_test_one        rshkill             vdom               
dd                  license             rshstat             version            
debug               lmgr_diag           rsm                 vfiler             
df                  lock                rtag                vif                
disk                log                 rtfo                vlan               
disk_fw_update      log_fio             sasadmin            vm_stat            
disk_list           logger              sasstat             vmservices         
disk_stat           logout              savecore            vol                
dns                 ls                  scsi                vol_db             
download            lun                 sectrace            vscan              
du                  mailbox             secureadmin         vstorage           
dump                man                 sesdiag             vtic               
dumpblock           maxfiles            setflag             waffinity_stats    
dumpstack           mbstat              setup               wafl               
echo                mem_scrub_stats     sftp                wafl_backdoor_stats
ems                 mem_stats           sh                  wafl_cmd_restrictions
environ             memerr              shelfchk            wafl_steal_stats   
environment         mkfile              show_faults         wafl_susp          
exit                mt                  showfh              wafltop            
export_stats        mv                  showfh4             wcc                
export_tbl_dump     nbtstat             signal              wrfile             
exportfs            ndmpcopy            sis                 xttcp              
fcadmin             ndmpd               sldiag              ypcat              
fcmon               ndp                 slist               ypgroup            
fcnic               netdiag             sm_mon              ypmatch            
fcp                 netmpstat           sm_mon_old          ypwhich  

Typing either “priv set” without specifying a privilege level, or “priv set admin” will take you back to the default admin privilege level.

mba-7m-1\*> priv set
mba-7m-1>

Command Syntax and Help

You can see the syntax for a command by passing it the “-?” flag.

mba-7m-1> arp -?
usage: arp [-n]
arp [-n] -a
arp -d
arp -s [temp] [pub]
arp -F

Even better, administrative-level commands have a full manual (man) page available explaining the operation of the command in detail:

mba-7m-1> man arp

na_arp(1) na_arp(1)

NAME
na_arp - Address resolution display and control

SYNOPSIS
arp [-n] hostname

arp [-n] -a

arp -d hostname

arp -s hostname ether_address [ temp ] [ pub ]

DESCRIPTION
The arp command displays and modifies the tables that the
address resolution protocol uses to translate between
Internet and Ethernet addresses.

With no flags, arp displays the current ARP entry for
hostname. The host may be specified by name or by number,
using Internet dot notation.

OPTIONS
-a Displays all of the current ARP entries.

-d Deletes an entry for the host called hostname.

-n IP addresses are displayed instead of hostnames.

-s Creates an ARP entry for the host called hostname
with the Ethernet address ether_address. The Eth-
ernet address is given as six hex bytes separated
by colons. The entry not will be permanent if the
words following -s includes the keyword temp. Tem-
porary entries that consist of a complete Internet
address and a matching Ethernet address are flushed
from the arp table if they haven't been referenced
in the past 20 minutes. A permanent entry is not
flushed.

If the words following -s include the keyword pub,
the entry will be "published"; that is, this system
will act as an ARP server, responding to requests
for hostname even though the host address is not
its own.

HA CONSIDERATIONS
In takeover mode, each node in an HA pair maintains its
own ARP table. You can make changes to the ARP table on
the live node, or you can make changes to the ARP table on
the failed node using the arp command in partner mode.
However, the changes you make in partner mode are lost
after a giveback.

VFILER CONSIDERATIONS
When run from a vfiler context, (for example, via the
vfiler run command), arp operates on the concerned vfiler.
As currently all vfilers in an ipspace share an arp table,
arp operates on the arp table of the concerned vfiler's
ipspace.

SEE ALSO
na_ifconfig(1), na_partner(1), na_ipspace(1),
na_vfiler(1), RFC1483.

6 June 1998 na_arp(1)

Command Completion

In Data ONTAP 7-mode, commands cannot be tab-completed in the shell, nor can they be abbreviated as with some other shells. Each command needs to be fully specified in order for it to be recognized.

Navigation and Editing

Command-line editing and navigation utilizes the standard keystrokes and combination previously discussed in CLI Efficiency: Common Basics

While you can navigate through your previously-entered commands using the up and down arrows, or Ctrl+n and Ctrl+p, there is no “history” command to simply display the contents of that history.

You can enter multiple commands on the same command line by separating each command with a semi-colon. The commands will then be executed in order of entry.

mba-7m-1> echo "help"; echo "I'm being held prisoner"; echo "in a terminal emulator"
help
I'm being held prisoner
in a terminal emulator

Updated 20141130: Fixed formatting that had somehow gotten broken since original publication.

Tech Smorgasbord #1

There’s always too much out there to learn and not remotely enough time to do it in. That’s always been the case, of course, but it just keeps getting worse – there’s all of the “old” stuff still to be learned (or re-learned) and then every minute of every day there’s more new, cool stuff that you want to dive into as well. There’s never enough time, so we need to try (somehow) to narrow our focus to the most interesting or most important – and hopefully there’s overlap between the two.

Here’s a few of the things that have popped onto my radar over the last week that I need to find some time to look into.


Vagrant Manager for OS X

A cool little addition to the toolbox for those using Vagrant (and Mac OS X). I’ve only dabbled with Vagrant to date but this looks like a good way to get myself using it even more.


Await – A modern implementation of Expect for Windows

This is a PowerShell implementation of Expect and is part of the PowerShell Resource Gallery – which has a ton of other modules and resources that should be investigated.

If you don’t know what Expect is you can read the Wikipedia page for background, but in a nutshell: it’s a program to provide automation of interactive applications. Expect this, do that. It’s a pretty handy tool, and I’m curious about this new implementation.


Etcd and Fleet

There have been a lot of configuration systems to pop up over the last few years, and Etcd (for /etc distributed) is one of these, brought to us by the folks at CoreOS – which is itself another project with which I need to become more conversant. I looked at CoreOS a few years back, but it’s come quite a long ways since then.


Hanlon (aka Razor 2.0)

The ever-prolific Nick Weaver and Tom McSweeney came out with Razor back in 2012 to provide “cloud provisioning” and it got quite a bit of attention at the time. Subsequently, Nick has moved on to other projects, and Tom renamed & relaunched the project as Hanlon. The project seems to be continuing to gather steam and deserves some more attention.


MidoNet

At the OpenStack Summit this week, Midokura announced that they would be open-sourcing their flagship network virtualization product, MidoNet. Midokura have been doing “software-defined networking” since before we began defining everything as SDx. I think I first heard of Midonet via a very positive Brad Hedlund blog post in 2012. Brad’s pretty well-known in the networking community, having made his name while working at Cisco before leaving for Dell, and now working for VMware’s NSX team.

Midonet was interesting technology even then, and open-sourcing that tech is an interesting move today: by doing so, they’re hoping to gain a leadership position in the OpenStack and open-source networking communities. Of course, there’s another vendor working to guide OpenStack networking (Cisco), and then there’s the little matter of the influence of Nicira in the OpenStack world. As we all know, Nicira is the foundation for what has become NSX, at the same time that VMware is also making forays into their own OpenStack distribution with VMware Integrated OpenStack.


SmartDataCenter and Manta

Joyent has been one of the more unique tech startups of the past few years, not least because they have eschewed building off of the typical Linux/FreeBSD base for their own SmartOS, which is based on the now-dead OpenSolaris. You may also know them as the originators of the popular Node.js platform. Joyent have a lot of frighteningly smart people working for them, including quite a few ex-Sun engineers such as Bryan Cantrill. They’ve built an interesting alternative cloud offering on SmartOS, and now they’ve released two of the other key pieces of that cloud as open-source. SmartDataCenter is effectively an OpenStack competitor providing cloud management and container orchestration, and Manta is an object storage system (built on top of ZFS).


 

Access Your NetApp Clustered Data ONTAP Logs From Your Browser

Getting logs from many systems can be a laborious and sometimes even painful process. You may have to view them semi-interactively from the command line. You may need to pull them off the system via FTP, SFTP, NFS, or some other protocol and then open them in a text editor (often after unzipping the downloaded files). Sometimes there’s a web GUI for viewing the logs, though this often provides only a display that can be copied/pasted & not the backing file(s) themselves.

In the past it’s been just as tedious to get logs off of NetApp FAS systems, but starting with clustered Data ONTAP 8.1.1 NetApp began providing web (HTTPS) access to easily download log files from the controllers. Initially, this access was not enabled by default and needed to be manually configured. Starting with 8.2.1 it is now enabled and accessible out of the box. It’s not the prettiest of interfaces, but it’s quite workable, and in addition to logs it also gives you access to core (crash) files if and when those occur.

When it’s enabled, all you need to do is navigate to the right URL and enter the appropriate credentials. The format of the URLs looks like this:

SPI_Login_Prompt

SPI_Logs

The service is accessed via the cluster management IP (or name – you can use the friendly DNS name and not IP), but the logs you’ll be going to are those on each individual node in the cluster. This is giving you access to far more logs than are available through the normal event logs displayed in OnCommand System Manager or via the CLI. In a later post, we’ll go through some of the more interesting and important log files you can access.

With Data ONTAP 8.2.1, the built-in admin account has access to this service by default. If you would like to create another user only for accessing these files (for a junior operator, for example) it’s a simple matter:

  • security login create -username <logviewer> -application http -authmethod password

And then enter the password for this new user account.


If you are running versions of clustered Data ONTAP between 8.1.1 – 8.2, here are the steps you’ll need to follow to enable this functionality:

  • vserver services web modify -vserver * -name spi -enabled true
    • This enables the SPI service on all vservers (only necessary for the cluster and node management vservers)
  • vserver services web access create -name spi -role admin -vserver <cluster-name>
    • This grants the admin role access to the SPI web service.
  • vserver services web access create -name compat -role admin -vserver <cluster-name>
    • This grants the admin role access to the compat web service.
    • In my testing, this setting was not strictly necessary for accessing the log files, but it is recommended by NetApp Support.

The official NetApp Knowledge Base article for setting up access is here: https://kb.netapp.com/support/index?page=content&id=1013814

 

Looking forward to NetApp Insight EMEA 2014

In a little more than one week I’ll be getting on a plane to head to Berlin for NetApp Insight EMEA 2014. I’ve never been to Europe, other than to the UK (does that *really* count as Europe?), so I’m quite looking forward to the experience. While I don’t expect to have a lot of time to sightsee, I definitely plan on exploring the city as much as I can. After all, how can one visit your ancestral homeland & not at least take a peek around?

This year the conference is being held November 17th – 20th at CityCube Berlin, and it marks the first conference for our European customers (following the first US conference for customers last week in Las Vegas). I expect to feel just as much energy and excitement from them as we did from the customers who attended Insight US in Las Vegas last week.

Whether you’re a customer, partner, or NetApp employee there will be a ton of opportunities to learn and things to do at Insight:

  • 22 Hand-on Labs
  • 7 on-site NetApp technical certification exams
    • NS0-145 NetApp Certified Storage Associate (NCSA)
    • NS0-170 FlexPod Design
    • NS0-155 NetApp Certified Data Administrator (NCDA) for Data ONTAP 7-Mode
    • NS0-159 NetApp Certified Data Administrator (NCDA) for clustered Data ONTAP
    •  Ns0-502 NetAp Certified Implementation Engineer – SAN for Data ONTAP 7-Mode
    • Ns0-504 NetAp Certified Implementation Engineer – SAN for clustered Data ONTAP
    • Ns0-510 NetAp Certified Implementation Engineer – Backup and Recovery
  • NetApp University Get Certified Prep Sessions – to help you prepare to take a NetApp certification exam
  • Insight Central – a solutions exchange populated with over 50 partner booths, NetApp booths, Media Hub, and more
  • Hundreds of Breakout Sessions
  • Plus General Sessions, networking events, parties, and more

NetAppInsightHoL

I’m lucky enough to attend the conference because I have a couple sessions of my own to present:

  • II-3-2000 – OpenStack for the Enterprise: FlexPod with Red Hat Enterprise Linux OpenStack Platform
    • Wednesday, Nov 19, 11:15 AM – 12:15 PM in M7 – Level 3
    • Thursday, Nov 20, 1:30 PM – 2:30 PM in M5 – Level 3
  • II-2-1931 – NetApp and Cisco Validated Designs for Service Providers and Large Enterprises
    • Wednesday, Nov 19, 8:30 AM – 9:30 AM in R13 – Level 3
    • Wednesday, Nov 19, 1:45 PM – 2:45 PM in M6 – Level 3

If you haven’t registered for Insight yet, or haven’t scheduled all of your sessions yet, there’s still time! Go here and get registered: https://netappinsight2014emea.activeevents.com/portal/newreg.ww

You can also get more information on the main website: http://www.netappinsight.com.

Whether you’re attending in person or not, be sure to follow the Twitter hashtag #NTAPinsight: we blew up the feed pretty well last week for the US conference and we’re going to do the same for Europe!

If you are attending Insight, please say hello – I’m looking forward to meeting many of you either at my sessions or at other points throughout the conference!