MSSQL to MySQL Data migration using MySQL workbench 6.3

Recently I was testing data migration from MSSQL to MySQL using MySQL Workbench. My aim was to include data with all datatype available in MSSQL for migration. In this following blog post will see data migration prerequisites, migration steps and few common errors.

About MySQL Workbench

MySQL Workbench is a unified visual tool for database architects, developers, and DBAs. MySQL Workbench provides data modeling, SQL development, and comprehensive administration tools for server configuration, user administration, backup, Data migration and much more. MySQL Workbench is available on Windows, Linux and Mac OS X.

When a supported RDBMS product is being migrated, the MySQL Workbench Migration Wizard will automatically convert as much information as it can, but you may still be required to manually edit the automatically migrated schema for difficult cases, or when the default mapping is not as desired.

Generally speaking, only table information and its data are automatically converted to MySQL. Code objects such as views, stored procedures, and triggers, are not. But supported RDBMS products will be retrieved and displayed in the wizard. You can then manually convert them, or save them for converting at a later time.

The following MS-SQL versions are currently tested and supported by the MySQL Workbench Migration Wizard.

Microsoft SQL Server 2000, 2005, 2008, 2012

Prerequisite

Download and install MySQL Workbench GUI tool.

The MySQL Workbench Migration Wizard uses ODBC to connect to a source database, except for MySQL. You will need the ODBC driver installed that corresponds to the database you want to migrate from.

Preparation

To be able to migrate from Microsoft SQL Server, ensure the following:

  • The source SQL Server instance is running, and accepts TCP connections.
  • You know the IP and port of the source SQL server instance. If you will be migrating using a Microsoft ODBC driver for SQL Server (the default in Windows), you will need to know the host and the name of the SQL Server instance.
  • Make sure that the SQL Server is reachable from where you will be running MySQL Workbench. More specifically, check the firewall settings.
  • Make sure that the user account has proper privileges to the database that will be migrated.

DATA SOURCE ODBC Configuration

In order to set up a connectivity between MSSQL and MySQL, We need to configure DATA SOURCE drivers ODBC with MS-SQL connection information.

Add new System data source.

odbc_pic

Required privileges on MS-SQL DB

CONNECT SQL

VIEW ANY DATABASE

VIEW ANY DEFINITION

If proper permission not given then at the migration it will throw warning as follow,

blog_lalit_pic2

DB Servers Data Types Mapping

Refer:

https://dev.mysql.com/doc/workbench/en/wb-migration-database-mssql-typemapping.html

Available datatype in MS-SQL:

https://msdn.microsoft.com/en-us/library/ms187752.aspx

Sample MS-SQL data

Sample table and data with different MS-SQL datatype

-- integer data types

CREATE TABLE dbo.int_table
(
MyBigIntColumn bigint
,MyIntColumn  int
,MySmallIntColumn smallint
,MyTinyIntColumn tinyint
);

ALter table int_table add MyBitColumn bit;

ALter table int_table add CONSTRAINT bit_def default 1  for MyBitColumn;

INSERT INTO dbo.int_table VALUES (9223372036854775807, 214483647,32767,255);

update int_table SET MyBitColumn=1;

Alter table int_table alter column MyBitColumn bit not null;

-- decimal and numeric Data datatypes

CREATE TABLE dbo.num_table
(
MyDecimalColumn decimal(5,2)
,MyNumericColumn numeric(10,5)
);

INSERT INTO dbo.num_table VALUES (123, 12345.12);

Alter table num_table add MyMoneycolumn money;

Alter table num_table add MysmallMoneycolumn smallmoney;

INSERT INTO num_table (MyMoneyColumn,MysmallMoneycolumn) values(20.098,45.68);

alter table num_table add c_real real,c_float float (32);

INSERT INTO num_table (c_real,c_float) values(2.0,43.67897);

-- datetime datatype

create table date_table(
ID               VARCHAR(4)         NOT NULL,
First_Name         VARCHAR(20),
Last_Name          VARCHAR(20),
Start_Date         DATE,
End_Date           DATE,
c_time             Time,
Salary             Money,
City               VARCHAR(20),
Description        VARCHAR(80),
c_datetime             datetime,
cs_smalldatetime    smalldatetime,
c_datetime2         datetime2
)

ALTER TABLE date_table ADD CONSTRAINT cc_datetime DEFAULT GETDATE() FOR c_datetime;

ALTER TABLE date_table ADD CONSTRAINT cc_time DEFAULT convert(time, getdate()) FOR c_time;

ALTER TABLE date_table ADD CONSTRAINT cc_startdate DEFAULT convert(date,getdate()) FOR start_date;

ALTER TABLE date_table ADD CONSTRAINT cc_enddate DEFAULT convert(date,getdate()) FOR end_date;

-- prepare data

insert into date_table(ID,  First_Name, Last_Name,Salary,  City, Description,cs_smalldatetime,c_datetime2)

values ('01','Jason',    'Martin', 1234.56, 'Toronto',  'Programmer','2007-05-08 12:35:00','2007-05-08 12:35:29. 1234567');

insert into date_table(ID,  First_Name, Last_Name,Salary,  City, Description,cs_smalldatetime,c_datetime2)

values('02','Alison',   'Mathews', 2234.78, 'Vancouver','Tester','2016-07-08 12:36:00','2006-07-08 12:36:29. 1234567'); 

-- char,varchar,nvarchar,tinyint,int,text datatypes

CREATE TABLE [dbo].[Employee_2](
[Id] [int] IDENTITY(1,1) NOT NULL,
[Name] [nvarchar](150) NULL,
[empid] [int] NOT NULL,
[age] [tinyint] NOT NULL,
[gender] VARCHAR(10) NOT NULL CHECK (gender IN('male', 'Female', 'Unknown'))
);

Alter table Employee_2 add primary key (Id);

-- Create a nonclustered index on a table or view

CREATE INDEX i1 ON dbo.Employee_2 (Name);

--Create a clustered index on a table and use a 3-part name for the table

CREATE CLUSTERED INDEX ci1 ON Employee_2([Id]);

CREATE UNIQUE INDEX pk1 ON dbo.Employee_2 (Id);

-- Create a nonclustered index with a unique constraint on 3 columns and specify the sort order for each column

CREATE UNIQUE INDEX ui1 ON dbo.Employee_2 (name DESC, empid ASC, age DESC);

INSERT INTO Employee_2 (Name,empid,age,gender) values ('lalit',268981,27,'male');

INSERT INTO Employee_2 (Name,empid,age,gender) values ('harsh',268982,28,'male');

INSERT INTO Employee_2 (Name,empid,age,gender) values ('jina',268983,27,'female');

INSERT INTO Employee_2 (Name,empid,age,gender) values ('xyz',268984,32,'Unknown');

ALTER table employee_2 add emp_notes2 text;

update employee_2 SET emp_notes2='test data Migration from mssql- mysql';

CREATE TABLE Persons
(
P_Id int NOT NULL,
Lastname varchar(40),
Firstname varchar(40) NOT NULL,
Address varchar(100),
City char(50),
PRIMARY KEY (P_Id)
);

INSERT INTO Persons values(1,'c','lalit','IT park','PUNE');

CREATE TABLE Orders
(
O_Id int NOT NULL,
OrderNo int NOT NULL,
P_Id int,
PRIMARY KEY (O_Id),
CONSTRAINT fk_PerOrders FOREIGN KEY (P_Id)
REFERENCES Persons(P_Id)
);

INsert INTO orders values(1,2234,1);

create table binary_table (c_binary binary , c_varbinary varbinary (max) ,c_image image);

INSERT INTO binary_table (c_varbinary)
values (convert(VARBINARY(max),44));

Few extra tables MS-SQL Table:

  1. [dbo].[Altiris, Inc_$Item] table with 20686 rows
CREATE TABLE [dbo].[Altiris, Inc_$Item](
[No_] [varchar](20) NOT NULL,
[Description] [varchar](100) NOT NULL,
[Description 2] [varchar](50) NULL,
[Blocked] [tinyint] NOT NULL,
[Last Date Modified] [datetime] NULL,
[Inactive] [tinyint] NOT NULL,
[Fixed Node] [tinyint] NOT NULL,
[Minimum Nodes] [int] NOT NULL,
[Maximum Nodes] [int] NOT NULL,
[Tax Group Code] [varchar](10) NOT NULL,
[Current Price List] [tinyint] NOT NULL,
[PrimeKey] [int] IDENTITY(1,1) NOT NULL,
CONSTRAINT [primeKey_FK1] PRIMARY KEY CLUSTERED
(
[PrimeKey] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

  1. [dbo].[Employee]
CREATE TABLE [dbo].[Employee](
[Id] [int] IDENTITY(1,1) NOT NULL,
[Name] [nvarchar](150) NULL,
CONSTRAINT [PK_Employee] PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

  1. [dbo].[CUSTOMERS]
CREATE TABLE [dbo].[CUSTOMERS](
[ID] [int] NOT NULL,
[NAME] [varchar](20) NOT NULL,
[AGE] [int] NOT NULL,
[ADDRESS] [char](25) NULL,
[SALARY] [decimal](18, 2) NULL,
PRIMARY KEY CLUSTERED
(
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

 
  1. [dbo].[Products]
CREATE TABLE [dbo].[Products](
[ProductID] [int] NOT NULL,
[ProductName] [varchar](25) NOT NULL,
[Price] [money] NULL,
[ProductDescription] [text] NULL,
[c_time] [datetime] NOT NULL,
PRIMARY KEY CLUSTERED
(
[ProductID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
ALTER TABLE [dbo].[Products] ADD  CONSTRAINT [DF_YourTable]  DEFAULT (getdate()) FOR [c_time]
GO

Note: Insert appropriate data in above tables for respective datatypes.

Migration using MySQL Workbench

mig1

MS-SQL Connection test:
mig2

NOTE:  Connection method ‘ODBC Data Source’ option will cause for following error. To avoid such error at the time of data migration use Connection method as ‘ODBC Data Source [FreeTDS]

mig3

MySQL Connection test:
mig4

Fetching schema:
mig5

Schema selection and DB naming settings:
mig6

Reverse Engineering:
mig7

Source object list:
mig9

Migration:
mig10

Manual editing: ( If needed)
mig11

NOTE: Will Fix these warnings in next steps.

Target creation options:
mig12

Create schemas:
mig13

Create target result:
mig14

As we have seen warnings for date_table as ‘Defaults value CONVERT(……) is not supported’. In MySQL we cannot use functions in with default, to tackle this problem will create trigger on table for insert operation as follows,

CREATE TRIGGER before_insert_date_table
before INSERT ON date_table
FOR EACH ROW
SET new.Start_Date = curdate() , new.End_Date= curdate() , new.c_time= curtime();

The DEFAULT value clause in a data type specification indicates a default value for a column. With one exception, the default value must be a constant; it cannot be a function or an expression. This means, for example, that you cannot set the default for a date column to be the value of a function such as NOW() or CURRENT_DATE. The exception is that you can specify CURRENT_TIMESTAMP as the default for TIMESTAMP and DATETIME columns.

https://dev.mysql.com/doc/refman/5.7/en/data-type-defaults.html

Another table that we need to fix is  Employee_2

MS-SQL Employee_2 table column definition:

[gender] VARCHAR(10) NOT NULL CHECK (gender IN('male', 'Female', 'Unknown'))

MySQL Employee_2 table column definition:
mig15

Appropriate datatype for ‘gender’ column, Let change it:

`gender` enum('Male','Female','Unknown') NOT NULL default 'Unknown',

Data transfer setup:
mig16

Bulk data transfer:
mig17

Data and Log analysis

Migration Report:

------------------------------------------------------------------------------------

MySQL Workbench Migration Wizard Report

Source: Microsoft SQL Server 12.0.4100

Target: MySQL 5.7.15

------------------------------------------------------------------------------------
-- Migration

-- Summary


Number of migrated schemas: 1

-- mysql_migration

Source Schema:   mysql_migration

- Tables:             11
- Triggers:           0
- Views:              4
- Stored Procedures:  3
- Functions:          0


2. Migration Issues
  - mysql_migration
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - CUSTOMERS
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - NAME
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - ADDRESS
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Products
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - ProductName
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - ProductDescription
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - c_time
    note  Default value is getdate(), so type was changed from DATETIME to TIMESTAMP
  - Employee
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Name
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Employee_2
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Name
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - gender
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - emp_notes
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - emp_notes2
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Persons
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Lastname
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Firstname
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Address
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - City
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Orders
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - int_table
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - MyBitColumn
    note  Source column type BIT was migrated to TINYINT(1)
  - num_table
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - date_table
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - ID
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - First_Name
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Last_Name
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Start_Date
    warning  Default value CONVERT([date],getdate(),0) is not supported
  - End_Date
    warning  Default value CONVERT([date],getdate(),0) is not supported
  - c_time
    warning  Default value CONVERT([time],getdate(),0) is not supported
  - City
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Description
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - c_datetime
    note  Default value is getdate(), so type was changed from DATETIME to TIMESTAMP
  - binary_table
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Altiris, Inc_$Item
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - No_
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Description
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Description 2
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci
  - Tax Group Code
    note  Collation SQL_Latin1_General_CP1_CI_AS migrated to utf8_general_ci


3. Object Creation Issues


4. Migration Details

4.1. Table mysql_migration.CUSTOMERS (CUSTOMERS)

Columns:
  - ID INT    
  - NAME VARCHAR(20)    
  - AGE INT    
  - ADDRESS CHAR(25)    
  - SALARY DECIMAL(18,2)    

Foreign Keys:

Indices:
  - PRIMARY (ID)


4.2. Table mysql_migration.Products (Products)

Columns:
  - ProductID INT    
  - ProductName VARCHAR(25)    
  - Price DECIMAL(19,4)    
  - ProductDescription LONGTEXT    
  - c_time TIMESTAMP  CURRENT_TIMESTAMP  

Foreign Keys:

Indices:
  - PRIMARY (ProductID)


4.3. Table mysql_migration.Employee (Employee)

Columns:
  - Id INT    
  - Name VARCHAR(150)    

Foreign Keys:

Indices:
  - PRIMARY (Id)


4.4. Table mysql_migration.Employee_2 (Employee_2)

Columns:
  - Id INT    
  - Name VARCHAR(150)    
  - empid INT    
  - age TINYINT UNSIGNED   
  - gender VARCHAR(10)    
  - emp_notes LONGTEXT    
  - emp_notes2 LONGTEXT    

Foreign Keys:

Indices:
  - PRIMARY (Id)
  - ci1 (Id)
  - i1 (Name)
  - pk1 (Id)
  - ui1 (Name, empid, age)


4.5. Table mysql_migration.Persons (Persons)

Columns:
  - P_Id INT    
  - Lastname VARCHAR(40)    
  - Firstname VARCHAR(40)    
  - Address VARCHAR(100)    
  - City CHAR(50)    

Foreign Keys:

Indices:
  - PRIMARY (P_Id)


4.6. Table mysql_migration.Orders (Orders)

Columns:
  - O_Id INT    
  - OrderNo INT    
  - P_Id INT    

Foreign Keys:
  - fk_PerOrders (P_Id) ON Persons (P_Id)

Indices:
  - PRIMARY (O_Id)


4.7. Table mysql_migration.int_table (int_table)

Columns:
  - MyBigIntColumn BIGINT    
  - MyIntColumn INT    
  - MySmallIntColumn SMALLINT    
  - MyTinyIntColumn TINYINT UNSIGNED   
  - MyBitColumn TINYINT(1)  1  

Foreign Keys:

Indices:


4.8. Table mysql_migration.num_table (num_table)

Columns:
  - MyDecimalColumn DECIMAL(5,2)    
  - MyNumericColumn DECIMAL(10,5)    
  - MyMoneycolumn DECIMAL(19,4)    
  - MysmallMoneycolumn DECIMAL(10,4)    
  - c_real FLOAT(24,0)    
  - c_float DOUBLE    

Foreign Keys:

Indices:


4.9. Table mysql_migration.date_table (date_table)

Columns:
  - ID VARCHAR(4)    
  - First_Name VARCHAR(20)    
  - Last_Name VARCHAR(20)    
  - Start_Date DATE    
  - End_Date DATE    
  - c_time TIME(6)    
  - Salary DECIMAL(19,4)    
  - City VARCHAR(20)    
  - Description VARCHAR(80)    
  - c_datetime TIMESTAMP  CURRENT_TIMESTAMP  
  - cs_smalldatetime DATETIME    
  - c_datetime2 DATETIME(6)    

Foreign Keys:

Indices:


4.10. Table mysql_migration.binary_table (binary_table)

Columns:
  - c_binary BINARY(1)    
  - c_varbinary LONGBLOB    
  - c_image LONGBLOB    

Foreign Keys:

Indices:


4.11. Table mysql_migration.Altiris, Inc_$Item (Altiris, Inc_$Item)

Columns:
  - No_ VARCHAR(20)    
  - Description VARCHAR(100)    
  - Description 2 VARCHAR(50)    
  - Blocked TINYINT UNSIGNED   
  - Last Date Modified DATETIME(6)    
  - Inactive TINYINT UNSIGNED   
  - Fixed Node TINYINT UNSIGNED   
  - Minimum Nodes INT    
  - Maximum Nodes INT    
  - Tax Group Code VARCHAR(10)    
  - Current Price List TINYINT UNSIGNED   
  - PrimeKey INT    

Foreign Keys:

Indices:
  - PRIMARY (PrimeKey)


II. Data Copy

  - `mysql_migration`.`Employee`
            Succeeded : copied 2 of 2 rows from [mysql_migration].[dbo].[Employee]            Succeeded : copied 2 of 2 rows from [mysql_migration].[dbo].[Employee]            Succeeded : copied 2 of 2 rows from [mysql_migration].[dbo].[Employee]            Succeeded : copied 2 of 2 rows from [mysql_migration].[dbo].[Employee]    
  - `mysql_migration`.`binary_table`
            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[binary_table]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[binary_table]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[binary_table]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[binary_table]    
  - `mysql_migration`.`num_table`
            Succeeded : copied 4 of 4 rows from [mysql_migration].[dbo].[num_table]            Succeeded : copied 4 of 4 rows from [mysql_migration].[dbo].[num_table]            Succeeded : copied 4 of 4 rows from [mysql_migration].[dbo].[num_table]            Succeeded : copied 4 of 4 rows from [mysql_migration].[dbo].[num_table]    
  - `mysql_migration`.`Products`
          error  `mysql_migration`.`Products`:malloc(1073741824) failed for blob transfer buffer          error  `mysql_migration`.`Products`:Failed copying 2 rows          error  `mysql_migration`.`Products`:malloc(1073741824) failed for blob transfer buffer          error  `mysql_migration`.`Products`:Failed copying 2 rows            Succeeded : copied 2 of 2 rows from [mysql_migration].[dbo].[Products]            Succeeded : copied 2 of 2 rows from [mysql_migration].[dbo].[Products]    
  - `mysql_migration`.`int_table`
            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[int_table]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[int_table]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[int_table]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[int_table]    
  - `mysql_migration`.`Orders`
            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[Orders]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[Orders]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[Orders]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[Orders]    
  - `mysql_migration`.`Employee_2`
            Succeeded : copied 4 of 4 rows from [mysql_migration].[dbo].[Employee_2]            Succeeded : copied 4 of 4 rows from [mysql_migration].[dbo].[Employee_2]            Succeeded : copied 4 of 4 rows from [mysql_migration].[dbo].[Employee_2]            Succeeded : copied 4 of 4 rows from [mysql_migration].[dbo].[Employee_2]    
  - `mysql_migration`.`CUSTOMERS`
            Succeeded : copied 6 of 6 rows from [mysql_migration].[dbo].[CUSTOMERS]            Succeeded : copied 6 of 6 rows from [mysql_migration].[dbo].[CUSTOMERS]            Succeeded : copied 6 of 6 rows from [mysql_migration].[dbo].[CUSTOMERS]            Succeeded : copied 6 of 6 rows from [mysql_migration].[dbo].[CUSTOMERS]    
  - `mysql_migration`.`date_table`
            Succeeded : copied 2 of 2 rows from [mysql_migration].[dbo].[date_table]            Succeeded : copied 2 of 2 rows from [mysql_migration].[dbo].[date_table]            Succeeded : copied 2 of 2 rows from [mysql_migration].[dbo].[date_table]            Succeeded : copied 2 of 2 rows from [mysql_migration].[dbo].[date_table]    
  - `mysql_migration`.`Persons`
            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[Persons]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[Persons]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[Persons]            Succeeded : copied 1 of 1 rows from [mysql_migration].[dbo].[Persons]    
  - `mysql_migration`.`Altiris, Inc_$Item`
            Succeeded : copied 20686 of 20686 rows from [mysql_migration].[dbo].[Altiris, Inc_$Item]

Validate data in MySQL database and All set !!

Advertisements

MySQL script automation and security

After MySQL installation, If you don’t have any enterprise level / any GUI interface for monitoring, backup then one of the option is, write your own scripts to automate these tasks.
In this Blog post, we are going to see few monitoring and backup scripts with covering common security issues.

Credential security

Following is a simple script, which will monitor MySQL service. In case MySQL service [mysqld] is down, then it will send email alert.

#!/bin/sh

# Connection details
MYSQL_USER="monitor"
MYSQL_PASS="lemon_pwd@123" ----> [# Plain text password, Security thread]
MYSQL_HOST="localhost"
SERVER_HOST=$( hostname )
EMAIL=user1@xyz.com,user2@xyz.com

# MySQL status
mysqladmin -u${MYSQL_USER} -p${MYSQL_PASS} -h${MYSQL_HOST} ping 2>/dev/null 1>/dev/null
if [ $? -ne 0 ]; then
echo "MySQL database is down " | mail -s " MySQL not running on $SERVER_HOST" "$EMAIL"
fi
# END!!

The Problem with above script is, it has mysql user credentials info in plain text. We should store mysql user credentials to somewhere safe in encrypted file.

mysql_config_editor and Credential security

The mysql_config_editor utility enables you to store authentication credentials in an encrypted login path file named .mylogin.cnf.

1. Create a user for monitoring with required privileges.

CREATE USER IF NOT EXISTS 'monitor'@'localhost' IDENTIFIED BY 'strong_pwd'; 

GRANT SELECT, RELOAD, PROCESS, REPLICATION CLIENT ON *.* TO 'monitor'@'localhost';

2. Store user credentials info into .mylogin.cnf (This conf file will get created under current OS user home directory)

shell> mysql_config_editor set --login-path=monitor_client --host=localhost --user=monitor --password                                                                                     

Enter password:<enter_mysql_monitor_user_password>

3. Verify the file contains

shell> mysql_config_editor print --login-path=monitor_client
[monitor_client]
user = monitor
password = *****
host = localhost

4. Implementation and use.

MySQL access with credentials:

 
shell> mysql -u monitor -h localhost -p  
  

MySQL access with mysql_config_editor login_path:

shell> mysql --login-path=minitor_client

Monitoring scripts

Shell script for MySQL service and replication monitoring.


#!/bin/bash

SERVER_HOST=$( hostname )
EMAIL=user1@xyz.com,user2@xyz.com

# MySQL service monitoring
mysqladmin --login-path=monitor_client ping 2>/dev/null 1>/dev/null
if [ $? -ne 0 ]; then
echo "MySQL database is down " | mail -s " MySQL not running on $SERVER_HOST" "$EMAIL"
fi

# Replication monitoring 

for MYSQL_HOST in localhost
do
  MSG1=`/usr/bin/mysql --login-path=monitor_client -e "show slave status\G;" | grep 'Slave_IO_Running:'`
  OUTPUT=(${MSG1//:/ })
  STATUS1=`echo ${OUTPUT[1]}`
  MSG2=`/usr/bin/mysql --login-path=monitor_client -e "show slave status\G;" | grep 'Slave_SQL_Running:'`
  OUTPUT=(${MSG1//:/ })
  STATUS2=`echo ${OUTPUT[1]}`
  if [ "$STATUS1" == "Yes" ] && [ "$STATUS2" == "Yes" ];
  then
    echo "$MYSQL_HOST = $STATUS1 - $STATUS2"
  else
    echo "$SERVER_HOST replication not working"
    mail -s "Replication DOWN - $SERVER_HOST" "$EMAIL" <<EOF
Please check $SERVER_HOST database replication.
EOF
  fi
done
# End!!

Backup scripts

1. Create a user for backup with required privileges.

CREATE USER IF NOT EXISTS 'backup'@'localhost' IDENTIFIED BY 'strong_pwd'; 

GRANT SELECT, RELOAD, SHOW DATABASES, LOCK TABLES,SHOW VIEW, EVENT, TRIGGER ON *.* TO 'backup'@'localhost';

2. Store backup user credentials info into .mylogin.cnf (This conf file will get created under current OS user home directory)

shell> mysql_config_editor set --login-path=backup_client --host=localhost --user=backup --password                                                                                          

Enter password:<enter_mysql_backup_user_password>

3. Backup script


#!/bin/bash

NOW="$(date +"%d-%m-%Y")"
BKP_DIR="/backups/full_backups/$NOW/"
SERVER_HOST=$( hostname )

mkdir -p "$BKP_DIR"
touch "$BKP_DIR/backup.log"

echo "Dumping MySQL databases into separate SQL command files into dir=$BKP_DIR" >> $BKP_DIR/backup.log

db_count=0

for d in $(mysql -NBA --login-path=backup_client -e 'show databases')
do
   if [[ "$d" != information_schema ]] && [[ "$d" != mysql ]] && [[ "$d" !=  performance_schema ]] && [[ "$d" !=  sys ]]; then
    (( db_count++ ))
   echo "DUMPING DATABASE: $d " >> $BKP_DIR/backup.log
mysqldump --login-path=backup_client --single-transaction  $d | gzip > $BKP_DIR/$d.sql.gz
echo "Dumping --triggers --routines --events for databases $d into dir=$BKP_DIR" >> $BKP_DIR/backup.log
mysqldump --login-path=backup_client --triggers --routines --events --no-create-info --no-data --no-create-db --skip-opt $d | gzip > $BKP_DIR/$d-routines.sql.gz
  fi
done

echo "$db_count databases dumped into dir=$BKP_DIR" >> $BKP_DIR/backup.log

find /apps/backups/full_backups/ -type d -ctime +6 -exec rm -rf {} \;

echo "OLDER than 6 days BACKUPS deleted successfully" >> $BKP_DIR/backup.log

# End!!

Automate these scripts runs and All set!!

MySQL service : Unable to setup unix socket lock file.

How to solve mysqld service restart problem for above error?

Problem :
I was adding shell and home directory for mysql user,executed following cmd,

shell> usermod -m -d /home/mysql -s /bin/bash mysql

If mysql is running and process running with mysql , we need to stop mysql otherwise it will throw an error like usermod: user mysql is currently used by process 27768

After stopping MySQL service and adding shell and homedir for mysql user, at the time mysqld service startup it started throwing error.


shell> service mysqld restart
Redirecting to /bin/systemctl restart mysqld.service
Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Shell> systemctl status mysqld.service

● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: deactivating (final-sigterm) (Result: exit-code) since Wed 2016-12-14 22:16:42 MST; 7min ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 35222 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=1/FAILURE)
Process: 35204 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 27768 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/mysqld.service
└─35226 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

mysqld error logs:

[ERROR] Could not create unix socket lock file /var/lib/mysql/mysql.sock.lock.
[ERROR] Unable to setup unix socket lock file.
[ERROR] Aborting

Tried to stop, kill mysqld service but still it’s not going out from process list.

Root cause : Suspecting change in process id after modifying mysql user properties.

Solution:

Just Move/Rename your my.cnf and start mysqld service with default configuration.You will see no more error at the time for service startup.
Move backuped up /renamed original my.cnf and restart mysqld service again.

service mysqld restart

And should start working fine, as it is.

ALl set!!