H730P + FreeNAS --> force mrsas driver

If you use FreeNAS with the H730P in HBA mode it will autoselect the old mfi driver. To force the correct mrsas driver add mrsas_load="YES" in /boot/loader.conf.
Also add in loader.conf
kern.geom.label.gptid.enable="0"
kern.geom.label.disk_ident.enable="0"
In /boot/device.hints add  hw.mfi.mrsas_enable="1"

secure/auth log Audit + shell history of all users to a central secured log

If you want to log everything everybody does and store it away securely do the following:

a) Install snoopy https://github.com/sensepost/Snoopy
b) setup a rsyslog server who writes a dedicated secure/auth log
c) configure rsyslog on each host to send secure/auth log info to the central rsyslog server

This will log even command history and you are also able to tell root users apart by session id.

sssd ldap error "cannot parse authtok"

If you ever come across this "cannot parse authtok" error in sssd, you most likely have a typo in your config:
ldap_default_authok --> ldap_default_authtok

[Zabbix] How to use Amazon SNS free (US only) SMS notification for Zabbix Monitoring

Amazon provides a notification service which also can send SMS. For US phone numbers the first 100 a month a free and this a nice thing to use for Zabbix monitoring.

To do that we'll need the following steps:

1. install aws-cli (http://docs.aws.amazon.com/cli/latest/userguide/installing.html)
2. create a Access Key ID and Secret Access Key
3. under SNS service create a new topic
4. create new subscription to that topic for each phone number you want to add
5. Protocol: SMS, Endpoint: your phone number
6. copy the Topic ARN
7. on a Debian system create /var/lib/zabbix and chown it for zabbix:zabbix
8. run: su -c 'aws configure' -s /bin/sh zabbix , and enter the both keys, then a region e.g. us-east-1 and the last leave empty. see http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
9. configure your action, under Operations of your Action, select Operation Type : Remote Command, select Execute on : Zabbix Server and enter following Command: aws sns publish --topic-arn arn:aws:sns:us-east-1:3062343543543:example --message "URGENT: INFRASTRUCTURE ISSUE"

[Zabbix] How to use active Agents

In the Agent config you have to have ServerActive=$zabbixip:10051 (it is not 10050!). Hostname=yourmachinehostname.

Unlink all templates from the host.

Then you have to clone zabbix agent template and OS linux template. Call them e.g. OS Linux Active. Go to each Item list, mark all items, go Mass Update, then change type to Zabbix Agent Active. The same for all discovery items.

Restart the Zabbix-Agent and wait a few minutes.

Unattented mysql installation under Debian for scripts

export DEBIAN_FRONTEND=noninteractive && apt-get -y -f install mysql-server-5.6 && mysqladmin -u root password XXXXXXXXX
#create mysql database and user
mysql -uroot -pXXXXXXXXX -e'create database YYYYYYYY;'
mysql -uroot -pXXXXXXXXX -e'GRANT ALL PRIVILEGES ON YYYYYYYY.* TO "user"@"localhost" IDENTIFIED BY "ZZZZZZZZZzz"';'

[Zabbix] How to monitor single processes

It is very simple to monitor processes. Just add a new template, or use any, create an item and a respective trigger for that item. The trigger should use trigger alarm on last value !=0. We will use exit 0 as all OK, exit 1 as issue!

Then put something like this in a .conf file in zabbix_conf.d/ directory

UserParameter=processes.postfix, if pgrep smtpd >/dev/null 2>&1; then echo 0 ;else echo 1; fi
UserParameter=processes.bind9, if pgrep named >/dev/null 2>&1; then echo 0 ;else echo 1; fi
UserParameter=processes.blacklist, if ps aux | grep [b]lacklist >/dev/null 2>&1; then echo 0 ;else echo 1; fi

[Zabbix] How to monitor Postfix data on Zabbix 3.0

1. Import template from: https://raw.githubusercontent.com/incognico/scripts/master/zabbix/postfix/smtp_and_postfix_passive_checks_zabbix_template.xml or http://pastebin.com/akwa89gG

2. Enable Remote Commands in Agent Config

3. In /etc/zabbix/zabbix_agentd.d/userparameter_postfix.conf :

UserParameter=postfix.pfmailq,mailq | grep -c '^[0-9A-Z]'
UserParameter=postfix[*],/usr/local/bin/postfix-zabbix-stats.bash $1

4. Install logtail and pflogsumm
5. crontab -e : */5 * * * * root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin && $(which logtail) /var/log/maillog > /tmp/logtailoffset && $(which pflogsumm) /tmp/logtailoffset -h 0 -u 0 --bounce_detail=0 --deferral_detail=0 --reject_detail=0 --no_no_msg_size --smtpd_warning_detail=0 > /tmp/pflogsummoffset && chown zabbix:zabbix /tmp/pflogsummoffset
6. Add /usr/local/bin/postfix-zabbix-stats.bash and chmod+x it.

 #!/bin/bash  
pfkey=$(echo "$1" | tr '_' ' ')

if [ "$pfkey" = "bytes received" ]; then
        number=$(cat /tmp/pflogsummoffset | grep -e 'bytes received' | cut -c1-7 | xargs)
        echo "$(($number *1024))"
elif [ "$pfkey" = "bytes delivered" ]; then
        number=$(cat /tmp/pflogsummoffset | grep -e 'bytes delivered' | cut -c1-7 | xargs)
        echo "$(($number *1024))"
else
        cat /tmp/pflogsummoffset | grep -E $pfkey | cut -c1-7 | head -1 | xargs
fi

[Zabbix] How to add hosts with curl and bash

This is for Zabbix 3.0:
The script executes as follows: addhost.sh 2.2.2.2 testhostname

 
 #!/bin/bash
 IP=$1  
 HOST_NAME=$2  
 # CONSTANT VARIABLES  
 ERROR='0'  
 ZABBIX_USER='Admin' #Make user with API access and put name here  
 ZABBIX_PASS='password' #Make user with API access and put password here  
 ZABBIX_SERVER='zabbix.server.net' #DNS or IP hostname of our Zabbix Server  
 API='https://zzabbix.server.net/api_jsonrpc.php'  
 HOSTGROUPID=6 #What host group to create the server in  
 TEMPLATEID=10001 #What is the template ID that we want to assign to new Servers?  
 # Authenticate with Zabbix API    
 authenticate() {  
         echo `curl -k -s -H 'Content-Type: application/json-rpc' -d "{\"jsonrpc\": \"2.0\",\"method\":\"user.login\",\"params\":{\"user\":\""${ZABBIX_USER}"\",\"password\":\""${ZABBIX_PASS}"\"},\"auth\": null,\"id\":0}" $API`  
     }   
 AUTH_TOKEN=`echo $(authenticate)|jq -r .result`   
 # Create Host  
 create_host() {  
         echo `curl -k -s -H 'Content-Type: application/json-rpc' -d "{\"jsonrpc\":\"2.0\",\"method\":\"host.create\",\"params\": {\"host\":\"$HOST_NAME\",\"interfaces\": [{\"type\": 1,\"main\": 1,\"useip\": 1,\"ip\": \"$IP\",\"dns\": \"\",\"port\": \"10050\"}],\"groups\": [{\"groupid\": \"$HOSTGROUPID\"}],\"templates\": [{\"templateid\": \"$TEMPLATEID\"}]},\"auth\":\"$AUTH_TOKEN\",\"id\":1}" $API`  
     }  
 output=$(create_host)  
 echo $output | grep -q "hostids"  
 rc=$?  
 if [ $rc -ne 0 ]  
  then  
      echo -e "Error in adding host ${HOST_NAME} at `date`:\n"  
      echo $output | grep -Po '"message":.*?[^\\]",'  
      echo $output | grep -Po '"data":.*?[^\\]"'  
      exit  
 else  
      echo -e "\nHost ${HOST_NAME} added successfully\n"  
      # start zabbix agent  
      #service zabbix-agent start  
      exit  
 fi  

How to export an LVM Volume via ssh if you have no local disk space

Host A has a 40GB Volume but not additional disk space to save the .img to, so you could later copy it somewhere.

You can do this:
dd if=/dev/volumegroup/volumename bs=64k | gzip -c | ssh root@10.0.0.1 'cat > /whereever/volumename.img.gz'

to uncompress the .img do: zcat volumename.img.gz > volumename.img

[Tutorial] How to resize (bigger) an LVM volume with an encrypted partition used by a VM

Here the version if your crypt device is sitting in a partition as in:

 xvdb                         202:16  0 550G 0 disk   
 --xvdb1                       202:17  0 550G 0 part   
 --luks-1d3d064f-7e4e-4653-99c1-0ef41f213819 (dm-0) 253:0  0 550G 0 crypt   

On Host
#extend to the FINAL size, if you want to add use +, see lvextend man
lvextend -L200G /dev/VGStorage/lvm-test-storage
On VM
umount /opt
cryptsetup luksHeaderBackup /dev/xvdb1 --header-backup-file /root/luksheader
cryptsetup luksClose luks-2892b120-247f-4e7f-834b-816cfb8baf68
fdisk -u /dev/xvdb
d
n
p
1
First Sector: 2048
Last Sector default
w
cat /proc/partitions, confirm number of blocks in partition as given by fdisk
cryptsetup luksOpen /dev/xvdb1 luks-2892b120-247f-4e7f-834b-816cfb8baf68
cryptsetup resize luks-2892b120-247f-4e7f-834b-816cfb8baf68
e2fsck -f /dev/mapper/luks-2892b120-247f-4e7f-834b-816cfb8baf68
resize2fs /dev/mapper/luks-2892b120-247f-4e7f-834b-816cfb8baf68

[Tutorial] How to resize (bigger) an LVM volume with encryption used as disk by a VM

Case:
Xen Host with Volume Group
1 Volume is used as xvdb by a VM and now you want to resize (bigger) the encrypted volume. This example is for ext2/3/4.
You can do this with running VM.

 xvdb                         202:16  0 550G 0 disk     
 --luks-1d3d064f-7e4e-4653-99c1-0ef41f213819 (dm-0) 253:0  0 550G 0 crypt   

On host:
#extending to 200G in total, use + to add, see man
lvextend -L200G /dev/VGStorage/examplevolume

On VM:
#umount first if not done yet
cryptsetup resize $mappeddevicename
e2fsck -f /dev/mapper/$mappeddevicename
resize2fs /dev/mapper/$mappeddevicename

$mappeddevicename is your mapped device name, which can be something like "secure" or if generated "luks-92342-..."

Fixing / Troubleshooting Linux Bridge

best source on this: http://www.microhowto.info/troubleshooting/troubleshooting_ethernet_bridging_on_linux.html

[tutorial] a primer to cdist

cdist is tool for configuration management, similar to puppy and ansible with the big difference that it does not need an agent on the target machine but only password-less ssh access. The whole thing is based on simple shell scripts and thus needs nearly nothing. Only the control server has 1 dependency: python

start here: git clone https://github.com/ungleich/cdist.git

Basically you have a manifest and types as the 2 most important things. The manifest says what to execute where and can be a simple one liner up to a complex file (see sample file).
Types are the "what is there to do" and can be interdependent, e.g. there is a type called __package which installs packages and automatically uses the correct package manager depending on what distro it detects (via an explorer). Now this type can be used in your type if you need to install dependencies you simply run e.g.

for package in nginx php5 mysql
          do __package $package
done

Another important thing is dependencies. If you have to make sure certain steps need to be done before other steps e.g. build-essential needs to be installed before you can compile something, then you have to use "require". Example"

require="__package/build-essential" __postgres_database $user --owner $user

Pay attention to spaces when you use require! Read manual about that

Now a type directory can contain the following files:

• manifest (optional)
• singleton (optional)
• explorer (optional)
• gencode (optional) 
• parameter (optional)

Types are stored below cdist/conf/type/. Their name should always be prefixed with two underscores (__) to prevent collisions with other executables in $PATH.

To implement a new type, create the directory cdist/conf/type/__NAME.
For more explanation of each see: http://www.nico.schottelius.org/software/cdist/man/latest/man7/cdist-type.html

Let's do an example:
We want to create a LEMP stack type:
we create new directory __LEMPstack

In the manifest we put:

#!/bin/sh
for package in nginx mysql-server php5-fpm php5-mysql
       __package $package
done

Now let us assume you want to change the nginx default port and have it reloaded. For that we can use the gencode-remote file, which executes after the other things.
We put the following:
echo "sed -i -e 's/80/8080/g' /etc/nginx/sites-enabled/default"
echo "service nginx reload"

Now to execute that whole thing you can simply put "__LEMPstack" in your manifest that is under /manifest/ (name doesn't matter, for a short one I usually just call it init) and run cdist like this: ./cdist config -v 192.128.100.100
Instead of IP you could also set any hostname if its defined in your manifest (see the sample file).
You will end up with an error about missing object_id. If you want to run types without object_id you need to create a "singleton" file in the type directory. Then it will work.

Best way to learn is to take a look at some of the types that cdist comes with.

Supersingular Isogeny Diffie-Hellman (SIDH) : Post-Quantum Curves

"We propose a new suite of algorithms that significantly improve the performance of supersingular isogeny Diffie-Hellman (SIDH) key exchange. Subsequently, we present a full-fledged implementation of SIDH that is geared towards the 128-bit quantum and 192-bit classical security levels. Our library is the first constant-time SIDH implementation and is more than 2.5 times faster than the previous best (non-constant-time) SIDH software. The high speeds in this paper are driven by compact, inversion-free point and isogeny arithmetic and fast SIDH-tailored field arithmetic: on an Intel Haswell processor, generating ephemeral public keys takes 51 million cycles for Alice and 59 million cycles for Bob while computing the shared secret takes 47 million and 57 million cycles, respectively. The size of public keys is only 751 bytes, which is significantly smaller than most of the popular post-quantum key exchange alternatives. Ultimately, the size and speed of our software illustrates the strong potential of SIDH as a post-quantum key exchange candidate and we hope that these results encourage a wider cryptanalytic effort."

source:Efficient algorithms for supersingular isogeny Diffie-Hellman https://eprint.iacr.org/2016/413.pdf
SIDH Library Download: http://research.microsoft.com/en-us/downloads/bd5fd4cd-61b6-458a-bd94-b1f406a3f33f/

Understanding and Hardening Linux Containers

A must read for container security: https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2016/april/ncc_group_understanding_hardening_linux_containers-10pdf

Xtrabackup innobackupex - restore script

 #!/bin/bash  
 service mysql stop  
 rm -r /var/lib/mysql/*  
 innobackupex --copy-back /path/to/backup  
 chown -R mysql:mysql /var/lib/mysql  
 service mysql start  
 echo "finished backup restore"  

Scaleway: Boot from disk

Everytime you reboot your server, you'll have to press Ctrl+B and exit iPXE to start the VPS from the local disk. Otherwise it will load a Linux kernel from the network and it obviously won't work. 

Scaleway: How to install OpenBSD or any other image

see here: https://community.scaleway.com/t/openbsd-on-a-vc1/1933

Scaleway: How to fix missing KVM and TUN module

Stop server, open advanced options further down, choose different boot script (usually called stable). It has the latest vanilla kernel with some slight scaleway changes. Upon boot just "modprobe kvm" and "modprobe tun"

When your Openvpn is not routing your webtraffic...minimal iptables openvpn config

iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE

A quick intro of Galera Cluster for Mariadb on Debian 8 Jessie

This will be very quick. Make sure you have this in  your my.cnf

[mysqld]
#mysql settings
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
query_cache_size=0
query_cache_type=0
bind-address=0.0.0.0
#galera settings
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="my_wsrep_cluster"
wsrep_cluster_address="gcomm://nodeA,nodeB,nodeC"
wsrep_sst_method=rsync

Adjust SST Method to whatever you prefer, read more here: http://galeracluster.com/documentation-webpages/sst.html

SST=State Transfer Method= The way of how to transfer the initial complete image of the databases.
IST= Incremental State Transfer = The replication stream across the nodes

Adjust Clustername and Node Adresses in gcomm line. You can, and I recommend to use hostnames (be sure to define them in /etc/hosts)! The reason for this is that other wsrep variables like "prefered donor" or "node name" expect a hostname and ip can cause issues (I learnt that the hard way).

Make sure following ports are open on all nodes:
http://support.severalnines.com/entries/22654676-firewall-ports

To start a new cluster you need a bootstrap node that acts as the initial starting point, yet later is just a simple node.
On Debian start a new cluster by running "galera_new_cluster" script.

MUST READ:
Monitoring the cluster:
http://galeracluster.com/documentation-webpages/monitoringthecluster.html

Now the best way to learn about all possible failure/shutdown scenarios is to read this: https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/

Also read up on all wsrep vars: https://mariadb.com/kb/en/mariadb/galera-cluster-system-variables/

You can read yours with running "SHOW STATUS LIKE 'wsrep%';"

Galera Failure Scenarios

Read this: https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/

When you have grant/permission issues related to 127.0.0.1/localhost

Ever wondered why e.g. MySQL denies access after you granted the user@127.0.0.1 permissions? That is because MySQL and others use a Unix Socket instead of TCP interface and thus make it a different case. So localhost does not always equal 127.0.0.1.

[Zabbix] How to setup the Zabbix Mysql Default Template

I did this on Zabbix 3.0 on Debian 8.3 Jessie
- First add the template to the host in the web interface and enable it.

- Second add following content to /etc/zabbix/zabbix_agentd.conf.d/userparameters_mysql.conf on the agent host:

UserParameter=mysql.status[*],echo "show global status where Variable_name='$1';" | HOME=/etc/zabbix mysql -N | awk '{print $$2}' UserParameter=mysql.size[*],echo "select sum($(case "$3" in both|"") echo "data_length+index_length";; data|index) echo "$3_length";; free) echo "data_free";; esac)) from information_schema.tables$([[ "$1" = "all" || ! "$1" ]] || echo " where table_schema='$1'")$([[ "$2" = "all" || ! "$2" ]] || echo "and table_name='$2'");" | HOME=/etc/zabbix mysql -N UserParameter=mysql.ping,HOME=/etc/zabbix mysqladmin -s ping | grep -c alive UserParameter=mysql.version,mysql -V

- Third, add following to /etc/zabbix/.my.cnf :

[client]
user=zabbix
password=yourpassword 

-  In MySQL run (adjust username/password):

GRANT USAGE ON *.* TO 'zabbix'@'127.0.0.1' IDENTIFIED BY '123456' 

- Restart Zabbix Agent, Data takes 1-2minute before it shows up on server

How to fix: non-responding redis-cli

So I installed redis. It was listening, I saw that in netstat -tulpn. But when I tried to use it or connect to redis-cli it was hanging.
Strace said:

...

stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3477, ...}) = 0
write(1, "2283:M 23 Feb 12:06:32.492 * The"..., 882283:M 23 Feb 12:06:32.492 * The server is now ready to accept connections on port 6379
) = 88
epoll_wait(3, {}, 10128, 0)             = 0
open("/proc/2283/stat", O_RDONLY)       = 6
read(6, "2283 (redis-server) R 2280 2280 "..., 4096) = 319
close(6) 

Solution: Make sure you allow local traffic in iptables if you drop the rest

iptables -A INPUT -s 127.0.0.1 -j ACCEPT

How to setup Drupal 7 for a High Traffic Website

The biggest issue with Drupal is that it uses the Database for caching and sessions.
So we change that, in my example I use redis, but all modules here work also with memcached & others.

a) Use Redis for Cache

see http://ixorthings.blogspot.com.es/2016/02/how-to-use-redis-on-all-drupal-7-cache.html

b) Use Redis for Sessions

Warning: Due to Drupals failure of having a open/standard session table the module might break other modules sessions! There is no fix for that. IF you run into issues try the native php management (see session proxy module man) which should work fine no matter what module.

Install session_proxy module

Add this to the settings.php

$conf['session_inc'] = 'sites/all/modules/contrib/session_proxy/session.inc';
$conf['session_storage_force_default'] = FALSE;
$conf['session_storage_class'] = 'SessionProxy_Storage_Cache';
$conf['session_storage_options']['cache_backend'] = 'Redis_Cache';

Pay attention to the module path!

How to use Redis on all Drupal 7 cache backends

Install Redis module.
Download zip from https://github.com/nrk/predis.
Unzip in /sites/all/libraries/
Configure Redis connection via Admin Menu

Put this in settings.php

1. define('PREDIS_BASE_PATH', DRUPAL_ROOT . '/sites/all/libraries/predis-1.0/'); $conf['redis_client_interface'] = 'Predis'; $conf['redis_client_host'] = 'YOURHOSTNAME/IP'; $conf['redis_client_port'] = 6379; $conf['cache_backends'][] = 'sites/all/modules/contrib/redis/redis.autoload.inc'; $conf['cache_default_class'] = 'Redis_Cache'; $conf['cache_class_cache'] = 'Redis_Cache'; $conf['cache_class_cache_form'] = 'Redis_Cache'; $conf['cache_class_cache_views'] = 'Redis_Cache'; $conf['cache_class_cache_page'] = 'Redis_Cache'; $conf['cache_class_cache_menu'] = 'Redis_Cache'; $conf['cache_class_cache_path'] = 'Redis_Cache'; $conf['cache_class_cache_entity_node'] = 'Redis_Cache'; $conf['cache_class_cache_entity_taxonomy_term'] = 'Redis_Cache'; $conf['cache_class_cache_entity_taxonomy_vocabulary'] = 'Redis_Cache'; $conf['cache_class_cache_entity_file'] = 'Redis_Cache'; $conf['cache_class_cache_entity_user'] = 'Redis_Cache'; $conf['cache_class_cache_filter'] = 'Redis_Cache'; $conf['cache_class_cache_admin_menu'] = 'Redis_Cache'; $conf['cache_class_cache_bootstrap'] = 'Redis_Cache'; $conf['lock_inc'] = 'sites/all/modules/contrib/redis/redis.lock.inc'; $conf['redis_client_base'] = 1;

Drupal 7 Hybridauth Issue Authentication failed! Facebook returned an invalid user id

If you run Drupal 7 with Hybridauth module on a HTTPS website and you encounter this Facebook issue about invalid user. Regarding Hybridauth version 2.14:

You have to edit the base_facebook.php and change "return http" to "return https" for the gethttpprotocol() function to enforce https use in case the detection for https doesn't work on your server.

You are welcome