Category Archives: Oracle

Logstash and Oracle – JDBC input settings

Lately, I began experimenting with elastic search, logstash and kibana. As a DBA, the first thing I wanted to try is Data base connectivity. Just like any new emerging technology, Oracle database is not the first priority on their implementation (and of course, documentation). There are much more examples for sexier (and free) databases like Postgresql or MySQL. As a result, there are only few examples and the configuaration is a real PITA.

While it is easy to find mysql database jdbc for logstash examples

INPUT {
  jdbc {
    jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"
    jdbc_user => "mysql"
    parameters => { "favorite_artist" => "Beethoven" }
    schedule => "* * * * *"
    statement => "SELECT * from songs where artist = :favorite_artist"
  }
}

It is much harder to find Oracle database jdbc logstash examples but I found something to start with at https://discuss.elastic.co/t/logstash-jdbc-input-oracle-settings/26996

jdbc {
      # The path TO our downloaded jdbc driver saved mine IN /bin, driver ojdbc6.jar also works. 
      #Best location can be /vendor folder
           jdbc_driver_library => "ojdbc7.jar"
      # ORACLE Driver Class
           jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
      # ORACLE jdbc connection string TO our DATABASE, ORACLE  jdbc:oracle:thin:@hostname:PORT/SERVICE
           jdbc_connection_string => "jdbc:oracle:thin:@hostname:PORT/SERVICE"
      # The USER AND password we wish TO EXECUTE our statement AS
           jdbc_user => "DBUSERNAME"
           jdbc_password => "DBPASSWORD"
 
    # our SQL query
    statement => "select * from AWESOMETABLE"
 
    #optional extras I USE 
      TYPE => "Database"
      tags => ["awesome", "import"]
 }

Since I did not have ojdbc7.jar on my computer I chose to use ojdbc6.jar so my Oracle database jdbc logstash query came out like this:

INPUT {
jdbc {
    jdbc_connection_string => "jdbc:oracle:thin:@\\my-host:1521:my-sid"
    jdbc_user => "db-user"
    jdbc_password => "db-pwd"
    jdbc_validate_connection => TRUE
    jdbc_driver_library => "C:\path\to\ojdbc6.jar"
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    statement => "SELECT * from my-table"
}
}
output {
elasticsearch {
INDEX => "contacts"
document_type => "contact"
document_id => "%{uid}"
hosts => "localhost:9200"
}
}

Along the way, I encountered two error messages that are the reason for this post:
The first

” Error: java::oracle.jdbc.driver.OracleDriver not loaded. Are you sure you’ve included the correct jdbc driver in :jdbc_driver_library?
You may be interested in the ‘–configtest’ flag which you can use to validate logstash’s configuration before you choose to restart a running system. ”

Running the query with the ‘–configtest’ flag did not help much but eventually, it turned out to be a simple typo. The oracle driver was not loaded only because the call was
” java::oracle.jdbc.driver.OracleDriver ” instead of ” Java::oracle.jdbc.driver.OracleDriver “.
(this one was really annoying but I hope this will save you time I spent on it ).

Another message I got along the way was

Java::JavaSql::SQLRecoverableException: IO Error: Invalid connection string fo
rmat, a valid format is: “host:port:sid”

apparently, while all examples for the connection string are in the following format:
jdbc:mysql://localhost:3306/mydb
The valid format in my case (oracle client 11 on windows 7) was
jdbc:oracle:thin:@\\my-host:1521:my-sid

This should get you connected but if you still get

Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
Check the following:
The database and the listener are running.
The specified port in the code is the same port that the database is listening .
There is no firewall blocking
Good luck

Can you move a table between schemas ?

Many reasons can raise the need to change a table owner or to move a table from one user (schema) to another . regardless of reason, one would expect that Oracle will provide an easy way to move a table to a different schema.

well, the short answer is NO . there is no direct method to change the table owner.

If you try something like

RENAME TABLE_NAME TO new_schema.table_name ;

Oracle Will return ORA-01765: specifying table’s owner name is not allowed
While you can workaround this error by omitting the schema name. However, in our case, this is exactly what we want to do so it is still a problem.

Similarly, the ALTER TABLE command will not do the work as well.

ALTER TABLE source_schema.table_name RENAME TO new_schema.table_name;

While a different error message (ORA-14047: ALTER TABLE|INDEX RENAME May Not Be Combined With Other Operations. ) is returned when trying to use the alter table command to move the table to a different schema, the result is the same.

Another option to move a table to a different user is to simply copy the table using

CREATE TABLE new_schema.table_name AS SELECT  * FROM  source_schema.table_name;

but this will generate a lot of I/O , redo logs and might take a lot of time so it can be a solution only for small tables.
Likewise, Using export and import will require downtime.

While there is no direct method to move a table to a different schema, you can use
One of the less known advantages of partitioning to do the task. the ability to exchange partitions between tables across schema can help us achieve the goal.
Following is the process to move a table from one schema to another:
1) Create a partitioned copy of the table – you do this by using

SELECT dbms_metadata.get_ddl('TABLE','TABLE2MOVE') FROM dual;

if you want just the plain DDL without all the storage definition etc. you can run

     BEGIN
       dbms_metadata.set_transform_param dbms_metadata.session_transform,'SEGMENT_ATTRIBUTES',FALSE);
      END;

remember to add a partition clause at the end of the ddl

 CREATE TABLE "NEW_SCHEMA"."TABLE2MOVE" 
   (	"ID" NUMBER, 
	"TEXT" VARCHAR2(100)
   ) 
   partition BY hash (id) (partition p1);

2) Exchange the partition between the source and the target table

ALTER TABLE NEW_SCHEMA.table2move exchange partition p1 WITH TABLE SOURCE_SCHEMA.table2move;

3) That’s it! verify it by selecting from the source and target tables

         SELECT * FROM NEW_SCHEMA.table2move;
         SELECT * FROM  SOURCE_SCHEMA.table2move;

One thing to remember is that partitioning is an extra cost option on top of the enterprise edition license. so, if you get ORA-00439: feature not enabled: Partitioning. the option is not enabled. verify it with

SELECT * FROM v$option;

monitor index usage (are my indexes being used at all?)

Indexes are the easiest solution for fast query performance improvement but the gain does not come without a cost. Indexes take up space and make inserts slower. In Addition, many times, the index is added to solve one slow query. If the query is not executed frequently, the performance improvement might not be worth the cost of making almost every insert and update slower. For example, even if the index is useful for a nightly or weekly report, the extra daily overhead might not be worth the few minutes it will save during the night.

therefore, it is important to know the if and how much the index is used. The tool for this job is index monitoring usage.

Basically the process is very simple. The first step is turning on monitoring since index monitoring usage is turned off by default. The next step is to let it run for a while. It is important to choose the monitoring period wisely as it supposed to correctly represent real life usage. After that you can turn off monitoring and query the usage view.

Turing on index monitoring for a specific index:

ALTER INDEX “YOUR-SCHMA-NAME”.”YOUR-INDEX-NAME” MONITORING USAGE;

If you get: ora-00054 resource busy and aquire with NOWAIT specified or timeout expired
This is because altering a table or an index requires an exclusive lock on the table therefore it will not work while DML operations are performed on the table

Turing on index monitoring for all indexes on all indexes for a specific schema:
declare
query_str varchar2(100);

begin

for i in ( select * from dba_indexes d where d.owner ='YOUR_SCHEMA' and index_type!='LOB' --and index_name='ASSIGNMENT_TERR'
) loop

query_str:='ALTER INDEX "' || i.owner || '"."' || i.index_name || '" NOMONITORING USAGE' ;
execute immediate query_str ;

end loop;

end;
Without adding the index_type!=’LOB’ you might get ORA-22864 cannot ALTER or DROP LOB indexes which means that you can not operate directly on a system-defined LOB index. You should perform operations on the corresponding LOB column.

Turing OFF index monitoring for a specific index:
ALTER INDEX "YOUR-SCHMA-NAME"."YOUR-INDEX-NAME" NOMONITORING USAGE;

You can analyze the results by running
select * from v$object_usage

several point to take into consideration:
don’t rush to drop the index if you see that it wasn’t used. It is possible that that it is used only once a month but it can still be crucial.

If you turn off monitoring and turning it on again, previous data is deleted. Therefore it might be wise to backup v$object_usage from time to time.

After you drop an index, the indesx data is deleted from v$object_usage

A simple workaround for bug 13477790 – XML/SQL query hangs after Oracle 11

IF you will run the following query on an oracle 11 db

select xmlelement("root",
(xmlelement("sub_root",
xmlforest('value' as "some_tag",
(select xmlagg(xmlelement("test",
xmlforest('code_value' as
"Code")))
from dual) as "SOME_AGG_CODES"))))

from dual

you might find that the sq/xml query simply hangs for many minutes after the upgrade. Many times, the following message will be displayed ” end-of-file on communication channel” . In other cases, it is possible that your session will be disconnected or might not be able to send a break message because you client will simply freeze.

This is even more problematic if this query was written on earlier versions of oracle like oracle 9. The same query will work without a problem on Oracle 9 or oracle and this can be very challenging if you encounter the problem after the actual upgrade to oracle 11.

Oracle identified this problem as a known issue bug #13477790 and released a patch with the same number. It appears that this patch solves this issue but applying a patch is not always easy on large production systems.
Another option for solving this issue is rewriting the the query using an additional xmlelement instead of the xmlforest but this can be time consuming as well.

A quick workaround for overcoming this bug without applying the 13477790 patch is adding a simple hint to the query. Adding the /*+NO_XML_QUERY_REWRITE*/ hint before the first element in the query. Will solve this issue.

select /*+NO_XML_QUERY_REWRITE*/ xmlelement("root",
(xmlelement("sub_root",
xmlforest('value' as "some_tag",
(select xmlagg(xmlelement("test",
xmlforest('code_value' as
"Code")))
from dual) as "SOME_AGG_CODES"))))

from dual

OCI-21700: object does not exist or is marked for delete (a quick reminder)

While many reasons can cause the OCI-21700: object does not exist or is marked for delete message, many times the OCI-21700 is caused by difficulty to access resources that are checked only at runtime.

The most common reason for this error is granting execute privileges on a function without granting privileges on referenced object like types for example.

Giving the same grants to the referenced objects will solve the problem.

Update: it appears that this error message can be raised when using Weblogic Data Source to run a function as well.
Sometimes, mostly after a network failure/database restart/DB schema changes, the “OCI-21700: object does not exist or is marked for delete” will be raised. The weird thing is that the same function will fail when running it with Weblogic but it can run without a problem when you run it from the sql console using the same credentials.
It appears that some Data Base actions like importing or droping objects or even unexpected network failure can “corrupt the DataSource”. If this is the case, all you have to do to solve the problem is restarting the Data Source.

Update 2: Apparently, there is a bug in Oracle 10 that raises the OCI-21700 error regardless of providing the required permission. A unexplained workaround, if all else fails, is creating the function with the same name in the schema you are connected to and retry. Even weirder, if this solves the problem, it will still work even if you drop the newly created function. Sound like Oracle voodoo but hey, if it solve the problem …