Testing World

Pink Ninja Tester

Author: pinkninjatester

Django: distinct

pinkninjatester

I know there is distinct() function in Django ORM. However, it is used differently for each type of database. In this case, my application is using PostgreSQL but our unit test is using SQLite. So yeah good luck with it.

Some example using distinct:

YourModel.objects.distinct() # distinct for all combinations of columns in the table
YourModel.objects.distinct("a_column") # distinct only for a specific column

But as I mentioned, that our unit test is using SQLite. And only the first example is working for other than PostgreSQL. If we use the second example for our sql lite, it will throw this error 

DISTINCT ON fields is not supported by this database backend

So, what should we do if we just want to do distinct for a specific column in other database other than PostgreSQL? Then we can use values() or value_list() such as:

YourModel.objects.values("a_column").distinct()

We probably think that it will be translated to 

SELECT DISTINCT "YourModel"."a_column" FROM "YourModel"

But surprisingly, it is not the case. So if YourModel has defined the ordering in its Meta. Then this ordering will be included in the select distinct query. Okay for example, YourModel has ordering to ‘modified’, then the modified will be included into the query such as below:

class YourModel(models.Model):
    a_column = models.TextField()
    modified = models.TextField()

    class Meta:
        ordering = [ 'modified', ]

YourModel.objects.values("a_column").distinct()

# will be translated to 
SELECT DISTINCT "YourModel"."a_column", "YourModel"."modified" FROM "YourModel" ORDER BY "YourModel"."modified"

So the solution/workaround is actually simple. We just need to add order_by with nothing on it. 

YourModel.objects.values("a_column").distinct().order_by()

# will be translated to 
SELECT DISTINCT "YourModel"."a_column" FROM "YourModel"

Happy coding!

Python: \ufeff solution

pinkninjatester

Have you ever had this problem when you open a csv file using python and it did not recognize the column name even though you are 3000% sure the column name is there. 

Example csv data which is save as my_example.csv

id;name;age
1;aaa;11
2;bbb;12
3;ccc;13

So then you tried to open it using python such as:

with open('my_example.csv') as csv_file:
    reader = csv.DictReader(csv_file, delimiter=";")
        for row in list(reader)[:5]:
            print(row['id')

But then you are surprised because it throws error “KeyError: id is not recognized on row[‘id’]”. So I changed the code from print(row[‘id’] to print(row). I did that just because I am curious how the ptyhon unpack the csv file. Surprisingly it shows like this below:

{'\ufeffid': '1', 'name': 'aaa', 'age': '11'}
{'\ufeffid': '2', 'name': 'bbb', 'age': '12'}
{'\ufeffid': '3', 'name': 'ccc', 'age': '13'}

What is this \ufeff ?????

So this is called BOM – Byte Order Mark and is used to tell the difference between big- and little-endian UTF-16 encoding.  And when this issue is most likely occur? Imagine you create a csv file via excel and then when you save it, you save it in UTF-8. Excel by default will put a signature on the file and afaik there is no way to avoid it. 

Oke, so it is encoding problem. And then we can solve it with adding encoding when we open the csv file. 

with open('my_example.csv', encoding='utf-8-sig') as csv_file:
    reader = csv.DictReader(csv_file, delimiter=";")
        for row in list(reader)[:5]:
            print(row['id')

And voila, it works again! But will it work if the file we are trying to open does not have this issue? It is! The utf-8-sig encoding will decode both utf-8-sig-encoded text and text encoded with the standard utf-8 encoding. 

Django: AttributeError – select_related

pinkninjatester 
AttributeError: 'YourObject' object has no attribute 'select_related'

If you received this error, first check how you build up your django query. I had this error because I mis-structured it. 

what I did:

MyModel.objects.filter(somequery=somevalue).select_related(myrelation)

But what I should do:

MyModel.objects.select_related(myrelation).filter(somequery=somevalue)

ORA-01435: user does not exist

pinkninjatester

I got this error when I tried to connect to Oracle within Apache Airflow. 

I am using docker oracle from gvenzl in M1 macbook. You can read the details on this page https://hub.docker.com/r/gvenzl/oracle-xe.

And in my db connection in Airflow, I defined my connection as

oracle://[username]:[pwd]@host.docker.internal:1521/XEPDB1?encoding=UTF-8&nencoding=UTF-8&threaded=False&events=False&mode=sysdba&purity=new&service_name=XEPDB1

XEPDB1 is the service name that is provided by gvenzl/oracle-xe by default.

In the beginning, I thought it was because the user I used does not exist or could not be found. So these are the step I did:

  1. I checked if the user exist. select * from all_users
  2. I found that the user does exist. So I did grant all to the user.
  3. I tried to connect to database via SQL Developer. And the connection is established!
  4. And various other things such as re-init the airflow and dockers, re-build, remove, etc. But it did not bring any result. 

In the end, I tried to read the airflow documentation again. And I saw that after port there should be no service name defined. 

AIRFLOW_CONN_ORACLE_DEFAULT='oracle://oracle_user:XXXXXXXXXXXX@1.1.1.1:1521?encoding=UTF-8&nencoding=UTF-8&threaded=False&events=False&mode=sysdba&purity=new'

So I tried it out by removing the service name defined there after port. And voila! It is working!

So, it seems that Airflow uses the wrong schema to Oracle. And of course Oracle usually use the same name for username and schema. The difference is schema is in capital and username is not in capital. 

So if the error message can be clearer, then it might help me to troubleshoot it faster. I was hoping that the error should be like

user or schema does not exist

Git: Origin vs Upstream

pinkninjatester

Upstream mostly refers to the original repo that you have forked.

Origin refers to the forked repository. Most often this is the repository that you cloned. (assuming that you forked the upstream and clone the forked repository instead of cloning directly the upstream)

When a repo is cloned, it has a default remote called origin that points to your fork on GitHub, not the original repo it was forked from.
To keep track of the original repo, you need to add another remote named upstream

so when we do git pull or git push then by default it will go to origin.

SQL – Insert non-existent value to a table from another table

pinkninjatester

Case study:

You have table A which has a column named ‘code’ and then you have another table called B which has a foreign key relationship with table A on column ‘code’. We want to select all rows in table A that table B does not have which in this case row bbb and ccc.

example: 

Table A:

CODE
aaa
bbb
ccc

Table B:

CODE
aaa

Solution:

SELECT A.CODE FROM A WHERE CODE NOT IN(SELECT CODE FROM B ) 

Flutter: Null Safety Solutions

pinkninjatester

Sometimes there is a dependency that does not conform to a null safety but we cannot do anything since it is in the dependency package. So what can we do to solve it?

First, we need to understand what null safety is. And then also understand what the problem will be. If you are sure that you want to run the app with unsound null safety that we can use the following command:

flutter run –no-sound-null-safety

OR 

you can configure your IDE to enable your app to run with unsound null safety.

In VS CODE search for flutter run additional args in your user settings and then add –no-sound-null-safety

  1. Go to Settings. In Mac you can find as in this picture.

2. Search for “Flutter run additional args” 

3. Then click “Add Item”

4. Now type “–no-sound-null-safety”

5. Click ok

Information about unsound null safety is in this link.

Install cocoapods for flutter dev

pinkninjatester

There is a time when you run ‘flutter doctor’ and then you got this problem below. 

[!] Xcode - develop for iOS and macOS
    ! CocoaPods 1.8.4 out of date (1.10.0 is recommended).
        CocoaPods is used to retrieve the iOS and macOS platform side's plugin
        code that responds to your plugin usage on the Dart side.
        Without CocoaPods, plugins will not work on iOS or macOS.
        For more info, see https://flutter.dev/platform-plugins
      To upgrade see
      https://guides.cocoapods.org/using/getting-started.html#installation for
      instructions.

It means that you need to install cocoapods. Well it is a pretty clear message right. And if you follow the link, you will be asked to execute this command below:

sudo gem install cocoapods

But when I executed this piece of command line, I got another error message which was:

ERROR:  Error installing cocoapods:
        ERROR: Failed to build gem native extension.

Solution:

brew cleanup -d -v 

brew install cocoapods

And hopefully you will not encountered any other issues while installing the cocoapods. In my case I need to do extra overwrite the link such as this:

brew link --overwrite cocoapods

Then tried to run the ‘flutter doctor’ again and check if there is no more error.