List directories via 'ls -d' - surprised me

List directories via 'ls -d' - surprised me

This was one of those feelings where I was left surprised with the humble linux command 'ls'

Here is what the man page says about ls -d -

-d, --directory

list directories themselves, not their contents
tried many times 'ls -d' all I got back was -

[sanjayu@dhcp-0-69:~]$
ls -dl
drwx--x---+ 110 sanjayu sanjayu 12288 Nov 10 19:39 .
[sanjayu@dhcp-0-69:~]$
ls -d
.

Now definitely there is something wrong, and I started searching the google. frankly not much came out as a help. I know we can do this via find ie find ./ -type d (using -maxdepth one can control recursive-ness). However, what intrigued me was man page and the contrary behaviour.

Finally found this - http://www.rapidtables.com/code/linux/ls.htm

ls -d */

Yum: search for a package through the disabled repos

Recently, I was searching for a package, it soon turned out that I need to archive this at least for myself. Here is it

Look for all the repos :

[stack@undercloud ~]$ yum repolist all
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
* base: mirror.raystedman.net
* extras: mirror.pac-12.org
* updates: mirrors.sonic.net
515 packages excluded due to repository priority protections
repo id repo name status
!base/7/x86_64 CentOS-7 - Base 9,363
!centos-ceph-jewel/7/x86_64 CentOS-7 - Ceph Jewel 82
!delorean delorean-python-networking-bigswitch-2dd6c2693b8b203b9ac2148e9747609161514af9 520+236
!delorean-newton-testing/x86_64 dlrn-newton-testing 830+515
!extras/7/x86_64 CentOS-7 - Extras 264
!rdo-qemu-ev/x86_64 RDO CentOS-7 - QEMU EV 70
!updates/7/x86_64 CentOS-7 - Updates 856
repolist: 11,985

Find a package in disabled repolist

#!/bin/bashset -ex

for i in `yum repolist all | awk '{print $1}' | cut -d '/' -f 1 | sed 's/!//'`;
do
yum --disablerepo="*" --enablerepo="${i}" search $1done

Breaking up the script

1. yum repolist all | awk '{print $1}' --> get the first columns and remove anything after '/'
2. sed 's/!//' remove the annoying ! at the beginning
3. search for the package use -

 yum --disablerepo="*" --enablerepo=$REPONAME search $PACKAGE_NAME

More here

Uploading to gist and sharing

Gist is a cool tool, here we are going to see how to use it from cli and use it as archiving or sharing utility.

Steps - 

• pip install python-gist
• create a personal token at github
• create a .gist file under $HOME with the following content

[gist]
token: 
editor: 

• Now you are set, use the following - 

gist create  - creates a new gist
gist edit    - edit the files in your gist
gist list    - prints a list of your gists
gist clone   - clones a gist
gist delete  - deletes a gist from github
gist files   - prints a list of the files in a gist
gist archive - downloads a gist and creates a tarball
gist content - prints the content of the gist to stdout
gist info    - prints detailed information about a gist
gist version - prints the current version

Skydive (real-time network topology and protocols analyzer) on Fedora

Skydive Installation on Fedora 25

Dependencies

• Go >= 1.5
• Elasticsearch >= 2.0
• libpcap
• libxml2
• protoc >= 3.0

On Fedora, Install steps are -

• Install GO
• wget -c https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz
• tar -C /usr/local -xzf go$VERSION.$OS-$ARCH.tar.gz
• add the following in /etc/profile or ~/.profile
  export PATH=$PATH:/usr/local/go/bin
• Install Elasticsearch
• Elasticsearch requires JDK, hence we will install openjdk, as it works with it
• sudo dnf install java-1.8.0-openjdk icedtea-web
• Download RPM from https://www.elastic.co/downloads/elasticsearch
• Install and start the services - 
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
• Install libpcap, libxml2
• dnf install gcc gcc-c++ glibc unzip libpcap libpcap-devel libxml2 libxml2-devel
• Install protoc (via building from source)
• wget -c https://github.com/google/protobuf/releases/download/v3.2.0/protoc-3.2.0-linux-x86_64.zip
• unzip protoc-3.2.0-linux-x86_64.zip
• cd protoc-3.2.0
• ./configure 
• make install
• Build skydive
• export GOPATH=`pwd1`
• mkdir -p $GOPATH/src/github.com/skydive-project
• git clone https://github.com/skydive-project/skydive.git $GOPATH/src/github.com/skydive-project/skydive
• cd $GOPATH/src/github.com/skydive-project/skydive
• make install

Running Skydive

• Now that skydive installed, check via running - skydive allinone. If this works, we are ok, we can configure skydive to run with our tripleo setup
• Create skydive.yml with the following content -

https://gist.github.com/adc0ded6c4d648cb9d1b531d51b90c1f

• start the analyzer via - skydive agent --conf /etc/skydive.yml
• start the agent  - skydive agent --conf /etc/skydive.yml
• Fire up browser and watch the url http://hostname:8082/

Running some test scenarios

• Download https://github.com/skydive-project/skydive/blob/master/scripts/simple.sh
• run it, sudo bin/simple.sh start 172.16.99.1/24 172.16.100.1/24 to create two VM over a bridge br-int. Look at the flows in UI, enjoy!

Final Results

UI screenshot -

Youtube video to mp3 music for your music player

I listen to a lot of music over youtube, and I feel, thats an awful waste of bandwidth.

If I could just get the MP3 downloaded and listened over and over. Well this is possible! Let me tell you how?


youtube-dl -https://github.com/rg3/youtube-dl

This is a Python code which is a downloader of the video - 

sudo wget https://yt-dl.org/downloads/latest/youtube-dl -O /usr/local/bin/youtube-dlsudo chmod a+rx /usr/local/bin/youtube-dlyoutube-dl https://www.youtube.com/watch?v=eKz6Jpg9op8

you can download a playlist as well in a folder and once the all are downloaded you can run this small shell script. Copy this in some file and run it

for i in *; do
f1=`sed 's/.\{4\}$//' <<< "$i"`;
echo ${f1}; f2="${f1}.mp3";
ffmpeg -i ${i} -q:a 0 -map a ${f2};
done

Docker based application on your laptop

You want to have apps running in docker container. why?

1. Its convenient, to upgrade and twea k the app independent of changes to your own OS. 
2. To have multiple images of upgrade apps, which you know are experimental and buggy.
3. Ship-able, so things can be shared in a way which is exactly reproduce-able 
4. Hype!

How to do it in the easiest of ways.

Git checkout - https://github.com/jfrazelle/dockerfiles

Someone has put a lot of hardwork and have made all the dockerfiles for most of the things you need - 

It is an exhaustive list.

Build it, hack it, resubmit it, use it!

I use the firefox, chrome, tor, slack containers. The sound is a bit buggy and doesnt work straight out of the box. Here are some blogs to get you started - Jassie Frazelle's Container on desktop, Setting the sound up.

Here is how I build and fired up chrome - 

#>cd ~/workspace

#>git clone https://github.com/jfrazelle/dockerfilescd

#>cd dockerfiles
#>docker build -t saneax/chrome-stable chrome/stable/

#>docker run -it --rm \

  --net host \ 

  --cpuset-cpus 0 \ 

  --memory 512mb \ 

  -v /tmp/.X11-unix:/tmp/.X11-unix:rw \ 

  -v /dev/snd:/dev/snd \ 

  --privileged \ 

  -v /dev/shm:/dev/shm \ 

  -v /etc/machine-id:/etc/machine-id \ 

  -v /var/lib/dbus:/var/lib/dbus \ 

  -v /run/user/`id -u`/pulse/native:/run/user/`id -u`/pulse/native \ 

  -v ~/.pulse:/home/$dockerUsername/.pulse \ 

  --name chrome saneax/chrome

Happy docking!

Create a customized KVM on laptop

What do I mean by saying I need a customized KVM?

1. A way to get the external DHCP IP address to the KVM
2. A qcow2 image modified maybe?
3. Multiple network nics with specific needs, ie over vlan, host only etc.
4. Use specific nic cards ie for DPDK reasons

etc etc.

The above are some of my reasons I moved out from Vagrant etc.
I have a Vagrant file which can pretty much use a yaml file to create a set of KVM's with some option to nics. But as usual more than that is required.

Here in I will state (for my own good) steps to do what exactly I want -

Network

I want two nics, one external bridged so that others on the LAN can access my VM directly via an IP (assuming the general LAN has a DHCP server with free IP addresses and is freely giving those IP addresses, if your administrators have mac binded the dhcp ip addresses, then the only way you can get this functionality is the SNAT way)

Disable Network Manager. How?

Create a manual Bridge via brctl and link with a physical network -

brctl addbr <bridge_name>
brctl addif <bridge_name> <phys_net -- connected to the local lan of the DHCP server>
ip link set <phys_net> up
ip link set <bridge_name> up
ifup <bridge_name>  (we will call this brex, for further use)

OR via ifcfg-

#> cat /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
TYPE=Bridge
DELAY=0
BOOTPROTO=dhcp
ONBOOT=yes
NM_CONTROLLED=no
PERSISTENT_DHCLIENT=yes
DHCLIENT_IGNORE_GATEWAY=no
GATEWAY=10.2.0.254
DNS1=10.2.0.26
DNS2=10.3.0.26

#> cat /etc/sysconfig/network-scripts/ifcfg-eno1
DEVICE="eno1"
ONBOOT=yes
UUID="f423eb24-2c1d-49d1-acdf-9a63ea867ff4"
IPV6INIT=no
BOOTPROTO=none
TYPE=Ethernet
NAME="eno1"
BRIDGE=br0

disable firewalls for bridges - These control whether or not packets traversing the bridge are sent to iptables for processing. In the case of using bridges to connect virtual machines to the network, generally such processing is *not* desired, as it results in guest traffic being blocked due to host iptables rules that only account for the host itself, and not for the guests.

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0

Define one host only network (this is virtual). DHCP server is required if you want to auto populate IP address. However generally having statically configured openstack nodes help.

(without DHCP)

cat > /tmp/hostonly.xml <<EOF
<network>
<name>hostonly</name>
<ip address="192.168.17.254" netmask="255.255.255.0"/>
</network>
EOF

(Load and autorestart, remember the bridge name is virbr0 ^^)

virsh net-define /tmp/hostonly.xml
virsh net-autostart hostonly
virsh net-start hostonly

virt-install pass the bridge for network - Below will set two nics inside your VM, 1st one should be the hostonly nic, can be used for provisioning openstack clusters. 2nd one is the external access nic.

virt-install <blah.. blah..> --network network:virbr0 --network network:brex (A complete virt-install is given below)

VM Setup

Check nested virtualization support and please enable it, docs here.

install all the required RPM's for this excercise -

yum install libvirt qemu-kvm virt-manager virt-install libguestfs-tools xorg-x11-apps xauth virt-viewer libguestfs-xfs -y

Get Image (centos - url to check latest qcow2 - Index of /centos/7/images )

Lets create the directories for our use

mkdir -p /vmimages/{qcow2-arch,qcow2-kvm}
cd /vmimages/qcow2-kvm/ (lets stay here and use all the commands from here)

Copy to a libvirt images folder ie 'cp CentOS-7-x86_64-GenericCloud.qcow2 /vmimages/qcow2-arch/'

wget -c 'http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2'
mv CentOS-7-x86_64-GenericCloud.qcow2 /vmimages/qcow2-arch/

Check the size of your qcow2 image disk (this is the max size of your VM disks, alas! this is not enough)

virt-filesystems --long -h --all -a ../qcow2-arch/CentOS-7-x86_64-GenericCloud.qcow2
Name       Type        VFS  Label  MBR  Size  Parent
/dev/sda1  filesystem  xfs  -      -    8.0G  -
/dev/sda1  partition   -    -      83   8.0G  /dev/sda
/dev/sda   device      -    -      -    8.0G  -

Lets increase it!, step 1. create a black qcow2 image of your desired size (remember these files are parse files, so when you create they wouldnt take all the space required)

qemu-img create -f qcow2 centos7.2-kvm1.qcow2 40G

Step 2. expand centos (downloaded qcow2 image) into the newly created 40G empty qcow2 image

virt-resize --expand /dev/sda1 ../qcow2-arch/CentOS-7-x86_64-GenericCloud.qcow2 centos7.2-kvm1.qcow2

check via virt-filesystems, of the new qcow2 image

virt-filesystems --long -h --all -a centos7.2-kvm1.qcow2
Name       Type        VFS  Label  MBR  Size  Parent
/dev/sda1  filesystem  xfs  -      -    40G   -
/dev/sda1  partition   -    -      83   40G   /dev/sda
/dev/sda   device      -    -      -    40G   -

If you want to use the awesome feature of KVM called 'Copy on Write', then some more steps are required. What it does, is it uses another disk for the VM to write the changes. So all the VM's can have one master disk and write the changes to there own copy disks.

qemu-img create -f qcow2 -b centos7.2-kvm1.qcow2 packstack-node1.qcow2 <-- choose a name of the backup file intutively so that it reflects the node you would be running.

Do some customization.. This has good docs here and more here!! (Here i start feeling the similarity with docker build)

Here we are removing cloud-init from the qcow2 image (its a nuisance, unless this image is being spawned in the cloud)

virt-customize -a packstack-node1.qcow2 --run-command 'yum remove cloud-init* -y'

Here we are setting a very complex password!

virt-customize -a packstack-node1.qcow2 --root-password password:Cent05

Finally! Create VM and start

We have come to the stage to start the KVM. ( --cpu SandyBridge,+vmx is to enable nested VM's. It is CPU architecture dependent and to be enabled in bios, more details here)

virt-install --ram 8096 --vcpus 4 --os-variant centos7.0 --disk path=/vmimages/qcow2-kvm/packstack-node1.qcow2,device=disk,bus=virtio,format=qcow2 --import --noautoconsole --vnc --network bridge:virbr1,model=e1000 --network bridge:br0,model=e1000 --cpu SandyBridge,+vmx --dry-run --print-xml --name packstacknode1 > packstacknode1.xml

Load your XML into libvirt

virsh define packstacknode1.xml
virsh start packstacknode1
virsh console packstacknode1

Some tyding.. login to console

create the ifcfg-eth{0,1} files

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=static
NM_CONTROLLED=no
IPADDR=192.168.17.101
NETMASK=255.255.255.0
EOF

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=dhcp
NM_CONTROLLED=no
DEFROUTE=yes
EOF              

put a hostname

hostnamectl set-hostname --static packstacknode1.redhat.local

Yay! you have a shiny new packstack node with e1000 ethernet nics, which supports ovs-dpdk
check 'ip a l' you should be able to connect to VM from your laptop on both the IP addresses.
From outside your laptop, you should be able to access the VM over LAN