t-head/thead-kernel
1
0
Fork 0
mirror of https://github.com/revyos/thead-kernel.git synced 2026-06-21 09:12:26 +02:00
Code Issues Packages Projects Releases Wiki Activity

driver: add npu-ax3386-gpl

Browse Source 
Add a new driver npu-ax3386-gpl, default compile as module.
The default generated modules are:
- vha.ko
- img_mem.ko
- vha_info.ko

Signed-off-by: Mingzheng Xing <xingmingzheng@iscas.ac.cn>
This commit is contained in:
Mingzheng Xing
2023-07-19 00:18:21 +08:00
committed by Han Gao
parent 6324bd2198
commit 9e49618e75
104 changed files with 62265 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
drivers/Kconfig
View File

@@ -8,6 +8,7 @@ source "drivers/eisa/Kconfig"
source "drivers/pci/Kconfig"
source "drivers/pcmcia/Kconfig"
source "drivers/rapidio/Kconfig"
source "drivers/nna/Kconfig"
source "drivers/base/Kconfig"

3
drivers/Makefile
View File

@@ -45,6 +45,9 @@ obj-$(CONFIG_VIRTIO) += virtio/
obj-$(CONFIG_VDPA) += vdpa/
obj-$(CONFIG_XEN) += xen/
# npu-ax3386-gpl driver
obj-y += nna/
# regulators early, since some subsystems rely on them to initialize
obj-$(CONFIG_REGULATOR) += regulator/

356
drivers/nna/GPLHEADER Normal file
View File

@@ -0,0 +1,356 @@
NOTE! This copyright does *not* cover user programs that use kernel
services by normal system calls - this is merely considered normal use
of the kernel, and does *not* fall under the heading of "derived work".
Also note that the GPL below is copyrighted by the Free Software
Foundation, but the instance of code that it refers to (the Linux
kernel) is copyrighted by me and others who actually wrote it.
Also note that the only valid version of the GPL as far as the kernel
is concerned is _this_ particular version of the license (ie v2, not
v2.2 or v3.x or whatever), unless explicitly otherwise stated.
Linus Torvalds
----------------------------------------
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.
51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Library General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
<signature of Ty Coon>, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Library General
Public License instead of this License.

27
drivers/nna/Kconfig Normal file
View File

@@ -0,0 +1,27 @@
menuconfig VHA
tristate "IMG neural network accelerator"
default m
if VHA
choice
prompt "VHA platform"
config VHA_THEAD_LIGHT_FPGA_C910
bool "driver runs with T-Head Light-FPGA hw platform using a device tree."
select HW_AX3
config VHA_DUMMY
bool "driver runs without hardware."
endchoice
config TARGET_OSID
int
default 0
config HW_AX3
bool "driver runs AX31xx/AX33xx/AX35xx hardware."
select VHA_MMU_MIRRORED_CTX
select VHA_SYS_AURA
config VHA_MMU_MIRRORED_CTX
bool "driver is compiled with mirrored MMU page tables regarding MODEL & IO."
depends on HW_AX3
config VHA_SYS_AURA
bool "driver runs with Aura system configuration file."
depends on HW_AX3
endif

7
drivers/nna/Makefile Normal file
View File

@@ -0,0 +1,7 @@
ifeq ($(CONFIG_VHA_NEXEF), y)
obj-$(CONFIG_VHA) += nexef_platform/
endif
obj-$(CONFIG_VHA) += vha/ img_mem/
obj-$(CONFIG_LOKI) += fenrir_loki/
subdir-ccflags-y += -I$(src)/include

8
drivers/nna/dmabuf_exporter/FindDmaBufExporter.cmake Normal file
View File

@@ -0,0 +1,8 @@
# Locate the userspace headers of dmabuf_exporter
#
get_filename_component(DMABUF_EXPORTER_PREFIX "${CMAKE_CURRENT_LIST_FILE}" PATH)
set (DMABUF_EXPORTER_FOUND TRUE)
set (DMABUF_EXPORTER_INCLUDE_DIR ${DMABUF_EXPORTER_PREFIX}/uapi)

36
drivers/nna/dmabuf_exporter/Makefile Normal file
View File

@@ -0,0 +1,36 @@
#
# Makefile for dmabuf exporter implementations
#
ifdef CONFIG_ION
obj-$(CONFIG_VHA) += dmabuf_exporter_ion.o
dmabuf_exporter_ion-objs := de_common.o de_heap_ion.o
dmabuf_exporter_ion-objs += de_heap_ion_example.o
# detect ION header in Linux Kernel tree
# srctree is needed here for kernels built with separate object dir (O=)
ifneq ($(wildcard $(srctree)/include/linux/ion.h),)
# some kernel trees have this non-standard path
ccflags-y += -DIMG_KERNEL_ION_HEADER="<linux/ion.h>"
else
# this is the default location
# the vanilla linux kernel does not export ion.h to include/linux
# adding -I to the entire directory would expose many internal header files
# so we use this somewhat ugly hack to use only this one with full path
# realpath is needed to expand full path, some kernel trees set srctree to .
ccflags-y += -DIMG_KERNEL_ION_HEADER="<$(realpath $(srctree))/drivers/staging/android/ion/ion.h>"
ccflags-y += -DIMG_KERNEL_ION_PRIV_HEADER="<$(realpath $(srctree))/drivers/staging/android/ion/ion_priv.h>"
endif
endif
ifdef CONFIG_DMA_SHARED_BUFFER
obj-$(CONFIG_VHA) += dmabuf_exporter_coherent.o
dmabuf_exporter_coherent-objs := de_common.o de_heap_coherent.o
obj-$(CONFIG_VHA) += dmabuf_exporter_noncoherent.o
dmabuf_exporter_noncoherent-objs := de_common.o de_heap_noncoherent.o
endif
ifdef CONFIG_GENERIC_ALLOCATOR
obj-$(CONFIG_VHA) += dmabuf_exporter_carveout.o
dmabuf_exporter_carveout-objs := de_common.o de_heap_carveout.o
endif

17
drivers/nna/dmabuf_exporter/README Normal file
View File

@@ -0,0 +1,17 @@
1. How to build
make -C /lib/modules/$(uname -r)/build M=/path/to/dmabuf_exporter modules
2. Carveout example
sudo insmod dmabuf_exporter_carveout.ko carveout_base=0x38000000 carveout_size=0x8000000
3. Coherent example
sudo insmod dmabuf_exporter_coherent.ko
3. Non-Coherent example
sudo insmod dmabuf_exporter_noncoherent.ko cache_type=3
cache_type: 1-cached, 2-uncached, 3-writecombine

176
drivers/nna/dmabuf_exporter/de_common.c Normal file
View File

@@ -0,0 +1,176 @@
/*
* de_common.h
*/
#include <linux/dma-buf.h>
#include <linux/err.h>
#include <linux/types.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
#include <linux/version.h>
#include <linux/miscdevice.h>
#include "uapi/dmabuf_exporter.h"
#include "de_heap.h"
/*
* Because this file is used in all modules, the kernel build system does
* not define KBUILD_MODNAME. This causes a build failure in kernels where
* dynamic debug is enabled, in all instances of pr_debug().
*
* dynamic debug messages for this file will use this name
*/
#ifndef KBUILD_MODNAME
#define KBUILD_MODNAME "dmabuf_exporter"
#endif
static struct miscdevice dmabuf_miscdevice;
static int de_file_open(struct inode *inode, struct file *file)
{
pr_debug("%s\n", __func__);
file->private_data = NULL;
return 0;
}
static int dmabuf_ioctl_create(struct file *file, unsigned long arg_size)
{
void *private_data;
size_t size;
int ret;
pr_debug("%s: private_data %p\n", __func__, file->private_data);
size = (arg_size + PAGE_SIZE - 1) & (~(PAGE_SIZE - 1));
pr_debug("%s: requested size %lu PAGE_SIZE %lu actual size %zu\n",
__func__, arg_size, PAGE_SIZE, size);
if (file->private_data) {
pr_err("%s: buffer already created!\n", __func__);
return -EBUSY;
}
ret = de_heap_buffer_create(size, PAGE_SIZE, &private_data);
if (ret)
return ret;
file->private_data = private_data;
return 0;
}
static int dmabuf_ioctl_export(struct file *file, unsigned long flags)
{
pr_debug("%s: private_data %p\n", __func__, file->private_data);
if (file->private_data)
return de_heap_export_fd(file->private_data, flags);
pr_err("%s: buffer has not been created!\n", __func__);
return -ENODEV;
}
static int de_file_release(struct inode *inode, struct file *file)
{
pr_debug("%s: private_data %p\n", __func__, file->private_data);
if (file->private_data) {
de_heap_buffer_free(file->private_data);
file->private_data = NULL;
}
return 0;
}
static long de_file_ioctl(struct file *file, unsigned int cmd,
unsigned long arg)
{
pr_debug("%s: cmd %x arg %lx\n", __func__, cmd, arg);
switch (cmd) {
case DMABUF_IOCTL_CREATE:
#ifdef CONFIG_COMPAT
case COMPAT_DMABUF_IOCTL_CREATE:
#endif
return dmabuf_ioctl_create(file, arg);
case DMABUF_IOCTL_EXPORT:
#ifdef CONFIG_COMPAT
case COMPAT_DMABUF_IOCTL_EXPORT:
#endif
return dmabuf_ioctl_export(file, arg);
default:
pr_err("%s: unknown cmd %x\n", __func__, cmd);
return -ENOTTY;
}
}
static const struct file_operations dmabuf_fops = {
.owner = THIS_MODULE,
.open = de_file_open,
.release = de_file_release,
.unlocked_ioctl = de_file_ioctl,
#ifdef CONFIG_COMPAT
.compat_ioctl = de_file_ioctl,
#endif
};
static int __init dmabuf_device_init(void)
{
int ret;
pr_info("%s\n", __func__);
dmabuf_miscdevice.minor = 128;
dmabuf_miscdevice.name = "dmabuf";
dmabuf_miscdevice.fops = &dmabuf_fops;
dmabuf_miscdevice.parent = NULL;
dmabuf_miscdevice.mode = 0666;
ret = misc_register(&dmabuf_miscdevice);
if (ret < 0) {
pr_err("%s: failed to register misc device %s\n",
__func__, dmabuf_miscdevice.name);
return ret;
}
pr_info("%s: registered misc device %s\n",
__func__, dmabuf_miscdevice.name);
ret = de_heap_heap_init();
if (ret < 0) {
misc_deregister(&dmabuf_miscdevice);
return ret;
}
return 0;
}
static void __exit dmabuf_device_deinit(void)
{
pr_info("%s\n", __func__);
de_heap_heap_deinit();
misc_deregister(&dmabuf_miscdevice);
}
module_init(dmabuf_device_init);
module_exit(dmabuf_device_deinit);
MODULE_AUTHOR("GPL");
MODULE_DESCRIPTION("DMA-BUF test driver");
MODULE_LICENSE("GPL v2");
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

28
drivers/nna/dmabuf_exporter/de_heap.h Normal file
View File

@@ -0,0 +1,28 @@
/*
* de_heap.h
*/
#ifndef DE_HEAP_H
#define DE_HEAP_H
#include <linux/types.h>
int de_heap_buffer_create(size_t size, unsigned long align,
void **private_data);
int de_heap_export_fd(void *private_data, unsigned long flags);
void de_heap_buffer_free(void *private_data);
int de_heap_heap_init(void);
void de_heap_heap_deinit(void);
#endif /* DE_HEAP_H */
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

468
drivers/nna/dmabuf_exporter/de_heap_carveout.c Normal file
View File

@@ -0,0 +1,468 @@
/*
* de_heap_carveout.c
*/
#include <linux/version.h>
#include <linux/types.h>
#include <linux/err.h>
#include <linux/vmalloc.h>
#include <linux/genalloc.h>
#include <linux/pci.h>
#include <linux/dma-buf.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
#include "de_heap.h"
/*
* module parameters
*/
static unsigned int use_pci = 0;
module_param(use_pci, uint, 0444);
MODULE_PARM_DESC(use_pci, "use PCI bar memory (default: false)");
static unsigned int cpu_map = 1;
module_param(cpu_map, uint, 0444);
MODULE_PARM_DESC(cpu_map, "map memory to CPU (default: true)");
/* mandatory carveout parameters (use_pci = 0) */
static unsigned long carveout_base = 0;
module_param(carveout_base, ulong, 0444);
MODULE_PARM_DESC(carveout_base, "physical base address. "
"mandatory when use_pci is false");
static unsigned long carveout_size = 0;
module_param(carveout_size, ulong, 0444);
MODULE_PARM_DESC(carveout_size, "physical size in bytes. "
"mandatory when use_pci is false");
/* mandatory pci parameters (use_pci = 1) */
static unsigned int pci_vendor = 0;
module_param(pci_vendor, uint, 0444);
MODULE_PARM_DESC(pci_vendor, "PCI vendor id. mandatory when use_pci is true");
static unsigned int pci_product = 0;
module_param(pci_product, uint, 0444);
MODULE_PARM_DESC(pci_product, "PCI product id. mandatory when use_pci is true");
static int pci_bar = -1;
module_param(pci_bar, int, 0444);
MODULE_PARM_DESC(pci_bar, "PCI bar index. mandatory when use_pci is true");
/* optional pci parameters (use_pci = 1) */
static unsigned long pci_size = 0;
module_param(pci_size, ulong, 0444);
MODULE_PARM_DESC(pci_size, "physical size in bytes. "
"used when use_pci is true. "
"when 0 (the default), use all memory in the PCI bar");
static unsigned long pci_offset = 0;
module_param(pci_offset, ulong, 0444);
MODULE_PARM_DESC(pci_offset, "offset from PCI bar start. "
"used when use_pci is true. optional (default: 0)");
static bool use_sg_dma = true;
module_param(use_sg_dma, bool, 0444);
MODULE_PARM_DESC(use_sg_dma,
"Sets sg_dma_address/len info");
/*
* internal values
*/
static phys_addr_t pool_base;
/* 12 bits (4096 bytes) */
#define POOL_ALLOC_ORDER 12
static struct gen_pool *heap_pool;
static struct pci_dev *pci_device;
struct buffer {
phys_addr_t phys;
size_t size;
struct sg_table *sg_table;
struct dma_buf *dma_buf;
dma_addr_t dma_base;
unsigned int dma_size;
};
/*
* dmabuf ops
*/
static struct sg_table *de_carveout_map_dma(struct dma_buf_attachment *attach,
enum dma_data_direction dir)
{
struct buffer *buffer = attach->dmabuf->priv;
pr_debug("%s\n", __func__);
if (use_sg_dma) {
sg_dma_address(buffer->sg_table->sgl) = buffer->dma_base;
sg_dma_len(buffer->sg_table->sgl) = buffer->dma_size;
}
return buffer->sg_table;
}
static void de_carveout_unmap_dma(struct dma_buf_attachment *attach,
struct sg_table *sgt,
enum dma_data_direction dir)
{
struct buffer *buffer = attach->dmabuf->priv;
pr_debug("%s\n", __func__);
if (use_sg_dma) {
sg_dma_address(buffer->sg_table->sgl) = (~(dma_addr_t)0);
sg_dma_len(buffer->sg_table->sgl) = 0;
}
}
static void de_carveout_release(struct dma_buf *buf)
{
struct buffer *buffer = buf->priv;
pr_info("%s phys address 0x%llx size %zu\n",
__func__, (unsigned long long)buffer->phys, buffer->size);
sg_free_table(buffer->sg_table);
kfree(buffer->sg_table);
gen_pool_free(heap_pool, buffer->phys, buffer->size);
kfree(buffer);
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,19,0)
static void *de_carveout_kmap_atomic(struct dma_buf *buf, unsigned long page)
{
pr_err("%s not supported\n", __func__);
return NULL;
}
#endif
static int de_carveout_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma)
{
struct buffer *buffer = dmabuf->priv;
pr_debug("%s\n", __func__);
if (!cpu_map) {
pr_err("%s not allowed (cpu_map is false)\n", __func__);
return -EIO;
}
vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
return remap_pfn_range(vma, vma->vm_start,
page_to_pfn(sg_page(buffer->sg_table->sgl)),
buffer->sg_table->sgl->length,
vma->vm_page_prot);
}
static void *de_carveout_kmap(struct dma_buf *dma_buf, unsigned long page)
{
struct buffer *buffer = dma_buf->priv;
void *ptr;
if (!cpu_map) {
pr_err("%s not allowed (cpu_map is false)\n", __func__);
return NULL;
}
ptr = (void __force *)ioremap(buffer->phys, buffer->size);
if (!ptr) {
pr_err("%s:carveout ioremap failed\n", __func__);
return NULL;
}
return ptr;
}
static void de_carveout_kunmap(struct dma_buf *buf, unsigned long page,
void *vaddr)
{
pr_debug("%s\n", __func__);
if (vaddr)
iounmap((void __iomem __force *)vaddr);
}
static void *de_carveout_vmap(struct dma_buf *buf)
{
return de_carveout_kmap(buf, 0);
}
static void de_carveout_vunmap(struct dma_buf *buf, void *kptr)
{
de_carveout_kunmap(buf, 0, kptr);
}
static const struct dma_buf_ops dmabuf_ops = {
.attach = NULL, /* optional */
.detach = NULL, /* optional */
.map_dma_buf = de_carveout_map_dma,
.unmap_dma_buf = de_carveout_unmap_dma,
.release = de_carveout_release,
.begin_cpu_access = NULL, /* optional */
.end_cpu_access = NULL, /* optional */
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,12,0)
.kmap_atomic = de_carveout_kmap_atomic,
.kunmap_atomic = NULL, /* optional */
.kmap = de_carveout_kmap,
.kunmap = de_carveout_kunmap, /* optional */
#else
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,19,0)
.map_atomic = de_carveout_kmap_atomic,
.unmap_atomic = NULL, /* optional */
#endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
.map = de_carveout_kmap,
.unmap = de_carveout_kunmap, /* optional */
#endif
#endif
.mmap = de_carveout_mmap,
.vmap = de_carveout_vmap,
.vunmap = de_carveout_vunmap,
};
int de_heap_buffer_create(size_t size, unsigned long align, void **private_data)
{
struct buffer *buffer;
struct dma_buf *dma_buf;
int ret;
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,1,0)
DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
#endif
pr_info("%s:carveout size %zu\n", __func__, size);
buffer = kzalloc(sizeof(struct buffer), GFP_KERNEL);
if (!buffer) {
pr_err("%s:carveout failed to allocate buffer\n", __func__);
return -ENOMEM;
}
buffer->phys = gen_pool_alloc(heap_pool, size);
if (!buffer->phys) {
pr_err("%s:carveout gen_pool_alloc failed for size %zu\n",
__func__, size);
ret = -ENOMEM;
goto free_buffer;
}
buffer->size = size;
buffer->sg_table = kzalloc(sizeof(struct sg_table), GFP_KERNEL);
if (!buffer->sg_table) {
pr_err("%s:carveout failed to allocate sg_table\n", __func__);
ret = -ENOMEM;
goto free_alloc;
}
ret = sg_alloc_table(buffer->sg_table, 1, GFP_KERNEL);
if (ret) {
pr_err("%s:carveout sg_alloc_table failed\n", __func__);
goto free_sg_table_mem;
}
sg_set_page(buffer->sg_table->sgl, pfn_to_page(PFN_DOWN(buffer->phys)),
PAGE_ALIGN(buffer->size), 0);
/* Store dma info */
buffer->dma_base = buffer->phys;
if (use_pci) {
buffer->dma_base -= pool_base;
buffer->dma_base += pci_offset;
}
buffer->dma_size = PAGE_ALIGN(size);
if (use_sg_dma) {
/* No mapping yet */
sg_dma_address(buffer->sg_table->sgl) = (~(dma_addr_t)0);
sg_dma_len(buffer->sg_table->sgl) = 0;
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(3,17,0)
dma_buf = dma_buf_export(buffer, &dmabuf_ops, size, O_RDWR);
#elif LINUX_VERSION_CODE < KERNEL_VERSION(4,1,0)
dma_buf = dma_buf_export(buffer, &dmabuf_ops, size, O_RDWR, NULL);
#else
exp_info.ops = &dmabuf_ops;
exp_info.size = size;
exp_info.flags = O_RDWR;
exp_info.priv = buffer;
exp_info.resv = NULL;
dma_buf = dma_buf_export(&exp_info);
#endif
if (IS_ERR(dma_buf)) {
pr_err("%s:carveout dma_buf_export failed\n", __func__);
ret = PTR_ERR(dma_buf);
goto free_sg_table;
}
buffer->dma_buf = dma_buf;
*private_data = buffer;
pr_info("%s:carveout phys address 0x%llx size %zu\n",
__func__, (unsigned long long)buffer->phys, buffer->size);
return 0;
free_sg_table:
sg_free_table(buffer->sg_table);
free_sg_table_mem:
kfree(buffer->sg_table);
free_alloc:
gen_pool_free(heap_pool, buffer->phys, buffer->size);
free_buffer:
kfree(buffer);
return ret;
}
void de_heap_buffer_free(void *private_data)
{
struct buffer *buffer = private_data;
pr_info("%s:carveout phys address 0x%llx size %zu\n",
__func__, (unsigned long long)buffer->phys, buffer->size);
dma_buf_put(buffer->dma_buf);
}
int de_heap_export_fd(void *private_data, unsigned long flags)
{
struct buffer *buffer = private_data;
struct dma_buf *dma_buf = buffer->dma_buf;
int ret;
pr_debug("%s:carveout %p\n", __func__, buffer);
get_dma_buf(dma_buf);
ret = dma_buf_fd(dma_buf, flags);
if (ret < 0) {
pr_err("%s:carveout dma_buf_fd failed\n", __func__);
dma_buf_put(dma_buf);
return ret;
}
pr_info("%s:carveout phys address 0x%llx export fd %d\n",
__func__, (unsigned long long)buffer->phys, ret);
return ret;
}
int de_heap_heap_init(void)
{
size_t pool_size;
int ret;
pr_debug("%s:carveout\n", __func__);
if (use_pci) {
unsigned long bar_base, bar_len;
if (pci_vendor == 0 || pci_product == 0 || pci_bar < 0) {
pr_err("%s:carveout missing pci parameters\n",
__func__);
return -EFAULT;
}
pci_device = pci_get_device(pci_vendor, pci_product, NULL);
if (pci_device == NULL) {
pr_err("%s:carveout PCI device not found\n", __func__);
return -EFAULT;
}
bar_base = pci_resource_start(pci_device, pci_bar);
if (bar_base == 0) {
pr_err("%s:carveout PCI bar %d not found\n",
__func__, pci_bar);
ret = -EFAULT;
goto free_pci_device;
}
bar_len = pci_resource_len(pci_device, pci_bar);
if (bar_len == 0) {
pr_err("%s:carveout PCI bar %d has zero length\n",
__func__, pci_bar);
ret = -EFAULT;
goto free_pci_device;
}
pr_info("%s:carveout PCI bar %d start %#lx length %ld\n",
__func__, pci_bar, bar_base, bar_len);
if (pci_size == 0)
pci_size = bar_len;
if (pci_offset + pci_size > bar_len) {
pr_err("%s:carveout pci_offset and size exceeds bar\n",
__func__);
ret = -EFAULT;
goto free_pci_device;
}
pool_base = bar_base + pci_offset;
pool_size = pci_size;
} else {
pci_device = NULL;
if (carveout_base == 0) {
pr_err("%s:carveout carveout_base not defined\n",
__func__);
return -EFAULT;
}
if (carveout_size == 0) {
pr_err("%s:carveout carveout_size not defined\n",
__func__);
return -EFAULT;
}
pool_base = carveout_base;
pool_size = carveout_size;
}
heap_pool = gen_pool_create(POOL_ALLOC_ORDER, -1);
if (!heap_pool) {
pr_err("%s:carveout gen_pool_create failed\n", __func__);
ret = -ENOMEM;
goto free_pci_device;
}
ret = gen_pool_add(heap_pool, (unsigned long)pool_base, pool_size, -1);
if (ret) {
pr_err("%s:carveout gen_pool_add failed\n", __func__);
goto free_pool;
}
pr_info("%s:carveout base %#llx size %zu\n", __func__,
(unsigned long long)pool_base, pool_size);
return 0;
free_pool:
gen_pool_destroy(heap_pool);
free_pci_device:
if (pci_device)
pci_dev_put(pci_device);
return ret;
}
void de_heap_heap_deinit(void)
{
pr_info("%s:carveout\n", __func__);
gen_pool_destroy(heap_pool);
if (pci_device)
pci_dev_put(pci_device);
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

303
drivers/nna/dmabuf_exporter/de_heap_coherent.c Normal file
View File

@@ -0,0 +1,303 @@
/*
* de_heap_coherent.c
*/
#include <linux/version.h>
#include <linux/dma-buf.h>
#include <linux/err.h>
#include <linux/types.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
#ifdef CONFIG_X86
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,12,0)
#include <asm/cacheflush.h>
#else
#include <asm/set_memory.h>
#endif
#endif /* CONFIG_X86 */
#include "de_heap.h"
#define MEMORY_ALLOCATION_FLAGS (GFP_HIGHUSER | __GFP_ZERO)
struct buffer {
size_t size;
void *vaddr;
struct sg_table *sg_table;
dma_addr_t handle;
};
/*
* dmabuf ops
*/
static void de_coherent_release(struct dma_buf *buf)
{
struct buffer *buffer = buf->priv;
pr_info("%s phys address 0x%llx\n",
__func__, (unsigned long long int)buffer->handle);
sg_free_table(buffer->sg_table);
kfree(buffer->sg_table);
#ifdef CONFIG_X86
set_memory_wb((unsigned long)buffer->vaddr,
(buffer->size + PAGE_SIZE - 1) / PAGE_SIZE);
#endif
dma_free_coherent(NULL, buffer->size, buffer->vaddr, buffer->handle);
kfree(buffer);
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,19,0)
static void *de_coherent_kmap_atomic(struct dma_buf *buf, unsigned long page)
{
pr_debug("%s\n", __func__);
return NULL;
}
#endif
static struct sg_table *de_coherent_map_dma(struct dma_buf_attachment *attach,
enum dma_data_direction dir)
{
struct buffer *buffer = attach->dmabuf->priv;
pr_debug("%s\n", __func__);
return buffer->sg_table;
}
static void de_coherent_unmap_dma(struct dma_buf_attachment *attach,
struct sg_table *sgt,
enum dma_data_direction dir)
{
pr_debug("%s\n", __func__);
}
static int de_coherent_mmap(struct dma_buf *dmabuf,
struct vm_area_struct *vma)
{
struct buffer *buffer = dmabuf->priv;
unsigned long user_count, count, pfn, off;
/*
* we could use dma_mmap_coherent() here, but it hard-codes
* an uncached behaviour and the kernel complains on x86 for
* a double mapping with different semantics (write-combine and
* uncached). Instead, we re-implement here the mapping.
* code copied from dma_common_mmap()
*/
pr_debug("%s\n", __func__);
user_count = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
count = PAGE_ALIGN(buffer->size) >> PAGE_SHIFT;
pfn = page_to_pfn(virt_to_page(buffer->vaddr));
off = vma->vm_pgoff;
if (off >= count || user_count > (count - off))
return ENXIO;
vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
return remap_pfn_range(vma, vma->vm_start, pfn + off,
user_count << PAGE_SHIFT,
vma->vm_page_prot);
}
static void *de_coherent_kmap(struct dma_buf *dma_buf, unsigned long page)
{
struct buffer *buffer = dma_buf->priv;
pr_debug("%s\n", __func__);
/* kernel memory mapping has been done at allocation time */
return buffer->vaddr;
}
static void de_coherent_kunmap(struct dma_buf *buf, unsigned long page,
void *vaddr)
{
pr_debug("%s\n", __func__);
}
static void *de_coherent_vmap(struct dma_buf *buf)
{
return de_coherent_kmap(buf, 0);
}
static void de_coherent_vunmap(struct dma_buf *buf, void *kptr)
{
de_coherent_kunmap(buf, 0, kptr);
}
static const struct dma_buf_ops dmabuf_ops = {
.attach = NULL, /* optional */
.detach = NULL, /* optional */
.map_dma_buf = de_coherent_map_dma,
.unmap_dma_buf = de_coherent_unmap_dma,
.release = de_coherent_release,
.begin_cpu_access = NULL, /* optional */
.end_cpu_access = NULL, /* optional */
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,12,0)
.kmap_atomic = de_coherent_kmap_atomic,
.kunmap_atomic = NULL, /* optional */
.kmap = de_coherent_kmap,
.kunmap = de_coherent_kunmap, /* optional */
#else
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,19,0)
.map_atomic = de_coherent_kmap_atomic,
.unmap_atomic = NULL, /* optional */
#endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
.map = de_coherent_kmap,
.unmap = de_coherent_kunmap, /* optional */
#endif
#endif
.mmap = de_coherent_mmap,
.vmap = de_coherent_vmap,
.vunmap = de_coherent_vunmap,
};
int de_heap_buffer_create(size_t size, unsigned long align, void **private_data)
{
struct buffer *buffer;
struct dma_buf *dma_buf;
int ret;
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,1,0)
DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
#endif
pr_info("%s:coherent size %zu\n", __func__, size);
buffer = kzalloc(sizeof(struct buffer), GFP_KERNEL);
if (!buffer) {
pr_err("%s:coherent failed to allocate buffer\n", __func__);
return -ENOMEM;
}
buffer->size = size;
buffer->vaddr = dma_alloc_coherent(NULL, size, &buffer->handle,
MEMORY_ALLOCATION_FLAGS);
if (!buffer->vaddr) {
pr_err("%s:coherent dma_alloc_coherent failed for size %zu\n",
__func__, size);
ret = -ENOMEM;
goto dma_alloc_coherent_failed;
}
#ifdef CONFIG_X86
set_memory_wc((unsigned long)buffer->vaddr,
(buffer->size + PAGE_SIZE - 1) / PAGE_SIZE);
#endif
buffer->sg_table = kzalloc(sizeof(struct sg_table), GFP_KERNEL);
if (!buffer->sg_table) {
pr_err("%s:coherent failed to allocate sg_table\n", __func__);
ret = -ENOMEM;
goto sg_table_malloc_failed;
}
ret = sg_alloc_table(buffer->sg_table, 1, GFP_KERNEL);
if (ret) {
pr_err("%s:coherent sg_alloc_table failed\n", __func__);
goto sg_alloc_table_failed;
}
sg_set_page(buffer->sg_table->sgl, virt_to_page(buffer->vaddr),
PAGE_ALIGN(size), 0);
#if LINUX_VERSION_CODE < KERNEL_VERSION(3,17,0)
dma_buf = dma_buf_export(buffer, &dmabuf_ops, size, O_RDWR);
#elif LINUX_VERSION_CODE < KERNEL_VERSION(4,1,0)
dma_buf = dma_buf_export(buffer, &dmabuf_ops, size, O_RDWR, NULL);
#else
exp_info.ops = &dmabuf_ops;
exp_info.size = size;
exp_info.flags = O_RDWR;
exp_info.priv = buffer;
exp_info.resv = NULL;
dma_buf = dma_buf_export(&exp_info);
#endif
if (IS_ERR(dma_buf)) {
pr_err("%s:coherent dma_buf_export failed\n", __func__);
ret = PTR_ERR(dma_buf);
goto dma_buf_export_failed;
}
dma_buf->priv = buffer;
*private_data = dma_buf;
pr_info("%s:coherent phys address 0x%llx virtual addr %p size %zu\n",
__func__, (unsigned long long int)buffer->handle,
buffer->vaddr, size);
return 0;
dma_buf_export_failed:
sg_free_table(buffer->sg_table);
sg_alloc_table_failed:
kfree(buffer->sg_table);
sg_table_malloc_failed:
#ifdef CONFIG_X86
set_memory_wb((unsigned long)buffer->vaddr,
(buffer->size + PAGE_SIZE - 1) / PAGE_SIZE);
#endif
dma_free_coherent(NULL, size, buffer->vaddr, buffer->handle);
dma_alloc_coherent_failed:
kfree(buffer);
return ret;
}
int de_heap_export_fd(void *private_data, unsigned long flags)
{
struct dma_buf *dma_buf = private_data;
struct buffer *buffer = dma_buf->priv;
int ret;
pr_debug("%s:coherent %p\n", __func__, dma_buf);
get_dma_buf(dma_buf);
ret = dma_buf_fd(dma_buf, flags);
if (ret < 0) {
pr_err("%s:coherent dma_buf_fd failed\n", __func__);
dma_buf_put(dma_buf);
return ret;
}
pr_info("%s:coherent phys address 0x%llx export fd %d\n",
__func__, (unsigned long long int)buffer->handle, ret);
return ret;
}
void de_heap_buffer_free(void *private_data)
{
struct dma_buf *dma_buf = private_data;
struct buffer *buffer = dma_buf->priv;
pr_info("%s:coherent phys address 0x%llx\n",
__func__, (unsigned long long int)buffer->handle);
dma_buf_put(dma_buf);
}
int de_heap_heap_init(void)
{
pr_info("%s:coherent\n", __func__);
return 0;
}
void de_heap_heap_deinit(void)
{
pr_info("%s:coherent\n", __func__);
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

212
drivers/nna/dmabuf_exporter/de_heap_ion.c Normal file
View File

@@ -0,0 +1,212 @@
/*
* de_heap_ion.c
*/
#include <linux/dma-buf.h>
#include <linux/err.h>
#include <linux/types.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
#include <linux/miscdevice.h>
#include "de_heap.h"
#include "de_heap_ion.h"
#if ((LINUX_VERSION_CODE >= KERNEL_VERSION(4, 4, 0)) && (LINUX_VERSION_CODE < KERNEL_VERSION(4, 5, 0)))
static struct ion_client *dmabuf_ion_client;
int de_heap_buffer_create(size_t size, unsigned long align, void **private_data)
{
struct ion_handle *ion_handle;
unsigned int heap_mask;
unsigned int heap_flags;
pr_info("%s:check %zu\n", __func__, size);
pr_debug("%s:ion size %zu\n", __func__, size);
heap_mask = de_heap_ion_get_heap_mask();
pr_info("%s: heap mask = %x\n", __func__, heap_mask);
heap_flags = de_heap_ion_get_heap_flags();
pr_info("%s: heap flags = %x\n", __func__, heap_flags);
pr_info("%s:ion_alloc dmabuf_ion_client = %lx, size = %i, , align = %i \n", __func__, (long unsigned int)dmabuf_ion_client, (int)size, (int)align);
ion_handle = ion_alloc(dmabuf_ion_client, size, align,
heap_mask, heap_flags);
if (IS_ERR_OR_NULL(ion_handle)) {
pr_err("%s:ion ion_alloc failed, ion_handle = %li\n", __func__, PTR_ERR(ion_handle));
if (IS_ERR(ion_handle)) {
return PTR_ERR(ion_handle);
} else {
return -ENOMEM;
}
} else {
pr_info("%s:ion handle %p size %zu\n", __func__, ion_handle, size);
}
*private_data = ion_handle;
return 0;
}
int de_heap_export_fd(void *private_data, unsigned long flags)
{
struct ion_handle *ion_handle = private_data;
int ret;
pr_debug("%s:ion\n", __func__);
ret = ion_share_dma_buf_fd(dmabuf_ion_client, ion_handle);
if (ret < 0) {
pr_err("%s:ion ion_share_dma_buf_fd failed\n", __func__);
return ret;
}
pr_info("%s:ion handle %p export fd %d\n", __func__, ion_handle, ret);
return ret;
}
void de_heap_buffer_free(void *private_data)
{
struct ion_handle *ion_handle = private_data;
pr_info("%s:ion handle %p\n", __func__, ion_handle);
ion_free(dmabuf_ion_client, ion_handle);
}
int de_heap_heap_init(void)
{
pr_info("%s:ion\n", __func__);
dmabuf_ion_client = de_heap_ion_create_ion_client();
pr_err("%s:dmabuf_ion_client = %li \n", __func__, (long)dmabuf_ion_client);
if (!dmabuf_ion_client) {
pr_err("%s:ion failed to get an ion client %lx \n", __func__, (long)dmabuf_ion_client);
return -EFAULT;
}
return 0;
}
void de_heap_heap_deinit(void)
{
pr_info("%s:ion\n", __func__);
de_heap_ion_destroy_ion_client(dmabuf_ion_client);
}
#elif ((LINUX_VERSION_CODE >= KERNEL_VERSION(4, 14, 0)) && (LINUX_VERSION_CODE < KERNEL_VERSION(4, 16, 0)))
int de_heap_heap_init(void)
{
pr_info("%s:ion\n", __func__);
return 0;
}
void de_heap_heap_deinit(void)
{
pr_info("%s:ion\n", __func__);
}
int de_heap_export_fd(void *private_data, unsigned long flags)
{
pr_info("%s:\n", __func__);
return 0;
}
#elif ((LINUX_VERSION_CODE >= KERNEL_VERSION(5, 4, 0)) && (LINUX_VERSION_CODE < KERNEL_VERSION(5, 5, 0)))
static struct ion_client *dmabuf_ion_client;
int de_heap_buffer_create(size_t size, unsigned long align, void **private_data)
{
struct ion_handle *ion_handle;
unsigned int heap_mask;
unsigned int heap_flags;
pr_info("%s:check %zu\n", __func__, size);
pr_debug("%s:ion size %zu\n", __func__, size);
heap_mask = de_heap_ion_get_heap_mask();
pr_info("%s: heap mask = %x\n", __func__, heap_mask);
heap_flags = de_heap_ion_get_heap_flags();
pr_info("%s: heap flags = %x\n", __func__, heap_flags);
pr_info("%s:ion_alloc dmabuf_ion_client = %lx, size = %i, , align = %i \n", __func__, (long unsigned int)dmabuf_ion_client, (int)size, (int)align);
ion_handle = ion_alloc(dmabuf_ion_client, size, align,
heap_mask, heap_flags);
if (IS_ERR_OR_NULL(ion_handle)) {
pr_err("%s:ion ion_alloc failed, ion_handle = %li\n", __func__, PTR_ERR(ion_handle));
if (IS_ERR(ion_handle)) {
return PTR_ERR(ion_handle);
} else {
return -ENOMEM;
}
} else {
pr_info("%s:ion handle %p size %zu\n", __func__, ion_handle, size);
}
*private_data = ion_handle;
return 0;
}
int de_heap_export_fd(void *private_data, unsigned long flags)
{
struct ion_handle *ion_handle = private_data;
int ret;
pr_debug("%s:ion\n", __func__);
ret = ion_share_dma_buf_fd(dmabuf_ion_client, ion_handle);
if (ret < 0) {
pr_err("%s:ion ion_share_dma_buf_fd failed\n", __func__);
return ret;
}
pr_info("%s:ion handle %p export fd %d\n", __func__, ion_handle, ret);
return ret;
}
void de_heap_buffer_free(void *private_data)
{
struct ion_handle *ion_handle = private_data;
pr_info("%s:ion handle %p\n", __func__, ion_handle);
ion_free(dmabuf_ion_client, ion_handle);
}
int de_heap_heap_init(void)
{
pr_info("%s:ion\n", __func__);
dmabuf_ion_client = de_heap_ion_create_ion_client();
pr_err("%s:dmabuf_ion_client = %li \n", __func__, (long)dmabuf_ion_client);
if (!dmabuf_ion_client) {
pr_err("%s:ion failed to get an ion client %lx \n", __func__, (long)dmabuf_ion_client);
return -EFAULT;
}
return 0;
}
void de_heap_heap_deinit(void)
{
pr_info("%s:ion\n", __func__);
de_heap_ion_destroy_ion_client(dmabuf_ion_client);
}
#else
#error "kernel not supported"
#endif
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

46
drivers/nna/dmabuf_exporter/de_heap_ion.h Normal file
View File

@@ -0,0 +1,46 @@
/*
* de_heap_ion.h
*
* platform specific interface for de_heap_ion
*
*/
#ifndef DE_HEAP_ION_H
#define DE_HEAP_ION_H
#include <linux/version.h>
/*
* gcc preprocessor defines "linux" as "1".
* [ http://stackoverflow.com/questions/19210935 ]
* IMG_KERNEL_ION_HEADER can be <linux/ion.h>, which expands to <1/ion.h>
*/
#undef linux
#include IMG_KERNEL_ION_HEADER
/*
* fetch the ion heap number (argument to ion_alloc)
*/
unsigned int de_heap_ion_get_heap_mask(void);
#if ((LINUX_VERSION_CODE >= KERNEL_VERSION(4, 14, 0)) && (LINUX_VERSION_CODE < KERNEL_VERSION(4, 16, 0)))
unsigned int de_heap_ion_get_heap_id_mask(void);
#endif
/*
* fetch the ion flags (argument to ion_alloc)
*/
unsigned int de_heap_ion_get_heap_flags(void);
/*
* fetch an ion client instance
*
* the implementation of this usually depends on ion_device (ion_client_create)
* which is platform specific
*/
struct ion_client *de_heap_ion_create_ion_client(void);
struct ion_client *de_heap_ion_destroy_ion_client(struct ion_client *dmabuf_ion_client);
#endif /* DE_HEAP_ION_H */

205
drivers/nna/dmabuf_exporter/de_heap_ion_example.c Normal file
View File

@@ -0,0 +1,205 @@
/*
* de_heap_ion_example.c
*/
#include "de_heap_ion.h"
#include <linux/sizes.h>
#include <linux/slab.h>
#include <linux/version.h>
#undef linux
#include IMG_KERNEL_ION_HEADER
#if ((LINUX_VERSION_CODE >= KERNEL_VERSION(4, 4, 0)) && (LINUX_VERSION_CODE < KERNEL_VERSION(4, 5, 0)))
#include IMG_KERNEL_ION_PRIV_HEADER
#endif
#define linux 1
#if ((LINUX_VERSION_CODE >= KERNEL_VERSION(4, 4, 0)) && (LINUX_VERSION_CODE < KERNEL_VERSION(4, 5, 0)))
static struct ion_device *idev;
static struct ion_client *client=NULL;
static struct ion_heap **heaps;
#endif
#if ((LINUX_VERSION_CODE >= KERNEL_VERSION(4, 4, 0)) && (LINUX_VERSION_CODE < KERNEL_VERSION(4, 5, 0)))
static void *carveout_ptr;
static void *chunk_ptr;
static struct ion_platform_heap dummy_heaps[] = {
{
.id = ION_HEAP_TYPE_SYSTEM,
.type = ION_HEAP_TYPE_SYSTEM,
.name = "system",
},
{
.id = ION_HEAP_TYPE_SYSTEM_CONTIG,
.type = ION_HEAP_TYPE_SYSTEM_CONTIG,
.name = "system contig",
},
{
.id = ION_HEAP_TYPE_CARVEOUT,
.type = ION_HEAP_TYPE_CARVEOUT,
.name = "carveout",
.size = SZ_4M,
},
{
.id = ION_HEAP_TYPE_CHUNK,
.type = ION_HEAP_TYPE_CHUNK,
.name = "chunk",
.size = SZ_4M,
.align = SZ_16K,
.priv = (void *)(SZ_16K),
},
};
static struct ion_platform_data dummy_ion_pdata = {
.nr = ARRAY_SIZE(dummy_heaps),
.heaps = dummy_heaps,
};
#endif
unsigned int de_heap_ion_get_heap_mask(void)
{
return 0x42;
}
#if ((LINUX_VERSION_CODE >= KERNEL_VERSION(4, 14, 0)) && (LINUX_VERSION_CODE < KERNEL_VERSION(4, 16, 0)))
unsigned int de_heap_ion_get_heap_id_mask(void)
{
return 1<<ION_HEAP_TYPE_SYSTEM;
}
#endif
unsigned int de_heap_ion_get_heap_flags(void)
{
return 0;
}
#if ((LINUX_VERSION_CODE >= KERNEL_VERSION(4, 4, 0)) && (LINUX_VERSION_CODE < KERNEL_VERSION(4, 5, 0)))
struct ion_client *de_heap_ion_create_ion_client(void)
{
int i, err;
pr_info("%s:\n", __func__);
/*
* usually involves fetching an ion_device from the system
* and calling ion_client_create()
*/
pr_info("%s:check ion_device_create \n", __func__);
idev = ion_device_create(NULL);
if (IS_ERR(idev)) {
pr_err("%s:ion ion_device_create failed %li \n", __func__, PTR_ERR(idev));
return (struct ion_client *)idev;
}
pr_info("%s:check kcalloc \n", __func__);
heaps = kcalloc(dummy_ion_pdata.nr, sizeof(struct ion_heap *),
GFP_KERNEL);
pr_info("%s:heaps = %lx \n", __func__, (long unsigned int)heaps);
if (!heaps) {
pr_err("%s:ion kcalloc heaps = %lx \n", __func__, (long unsigned int)heaps);
ion_device_destroy(idev);
return ERR_PTR(-ENOMEM);
}
/* Allocate a dummy carveout heap */
carveout_ptr = alloc_pages_exact(dummy_heaps[ION_HEAP_TYPE_CARVEOUT].size,
GFP_KERNEL);
if (carveout_ptr) {
dummy_heaps[ION_HEAP_TYPE_CARVEOUT].base = virt_to_phys(carveout_ptr);
} else {
pr_err("ion_dummy: Could not allocate carveout\n");
}
/* Allocate a dummy chunk heap */
chunk_ptr = alloc_pages_exact(dummy_heaps[ION_HEAP_TYPE_CHUNK].size,
GFP_KERNEL);
if (chunk_ptr) {
dummy_heaps[ION_HEAP_TYPE_CHUNK].base = virt_to_phys(chunk_ptr);
} else {
pr_err("ion_dummy: Could not allocate chunk\n");
}
for (i = 0; i < dummy_ion_pdata.nr; i++) {
struct ion_platform_heap *heap_data = &dummy_ion_pdata.heaps[i];
if (heap_data->type == ION_HEAP_TYPE_CARVEOUT && !heap_data->base) {
pr_info("ion_dummy: ION_HEAP_TYPE_CARVEOUT skipped heap_data->base == %lx \n", heap_data->base);
continue;
}
if (heap_data->type == ION_HEAP_TYPE_CHUNK && !heap_data->base) {
pr_info("ion_dummy: ION_HEAP_TYPE_CHUNK skipped heap_data->base == %lx \n", heap_data->base);
continue;
}
heaps[i] = ion_heap_create(heap_data);
if (IS_ERR_OR_NULL(heaps[i])) {
pr_info("ion_dummy: ion_heap_create failed, returned = %lx, for heap id = %d \n", (unsigned long)(heaps[i]), i);
err = PTR_ERR(heaps[i]);
goto err;
}
ion_device_add_heap(idev, heaps[i]);
}
pr_info("%s:ion ion_device_create success idev = %lx \n", __func__, (long)idev);
client = ion_client_create(idev, "ion_client");
if (IS_ERR_OR_NULL(client)) {
pr_info("%s:ion ion_client_create failed idev = %lx client = %li\n", __func__, (long)idev, PTR_ERR(client));
ion_device_destroy(idev);
return (client);
}
return client;
err:
for (i = 0; i < dummy_ion_pdata.nr; ++i)
ion_heap_destroy(heaps[i]);
kfree(heaps);
if (carveout_ptr) {
free_pages_exact(carveout_ptr,
dummy_heaps[ION_HEAP_TYPE_CARVEOUT].size);
carveout_ptr = NULL;
}
if (chunk_ptr) {
free_pages_exact(chunk_ptr,
dummy_heaps[ION_HEAP_TYPE_CHUNK].size);
chunk_ptr = NULL;
}
ion_device_destroy(idev);
return ERR_PTR(err);
return client;
}
/*
* returns error code or success (0)
*/
struct ion_client *de_heap_ion_destroy_ion_client(struct ion_client *dmabuf_ion_client)
{
pr_info("%s:de_heap_ion_destroy_ion_client \n", __func__);
if (IS_ERR(idev) || idev == NULL) {
pr_err("%s:ion device not present idev = %li \n", __func__, (idev==NULL) ? (long)NULL : PTR_ERR(idev));
return (struct ion_client *)(idev);
}
if (dmabuf_ion_client>0) {
ion_client_destroy(dmabuf_ion_client);
}
ion_device_destroy(idev);
idev = NULL;
return NULL;
}
#elif ((LINUX_VERSION_CODE >= KERNEL_VERSION(4, 14, 0)) && (LINUX_VERSION_CODE < KERNEL_VERSION(4, 16, 0)))
#else
#error "Linux kernel not supported"
#endif
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

496
drivers/nna/dmabuf_exporter/de_heap_noncoherent.c Normal file
View File

@@ -0,0 +1,496 @@
/*
* de_heap_noncoherent.c
*/
#include <linux/version.h>
#include <linux/dma-buf.h>
#include <linux/err.h>
#include <linux/types.h>
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
#include <linux/vmalloc.h>
#ifdef CONFIG_X86
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,12,0)
#include <asm/cacheflush.h>
#else
#include <asm/set_memory.h>
#endif
#endif /* CONFIG_X86 */
#include "de_heap.h"
#define MEMORY_ALLOCATION_FLAGS (GFP_DMA32 | __GFP_ZERO)
enum mem_cache_type {
MEM_TYPE_CACHED = 1,
MEM_TYPE_UNCACHED = 2,
MEM_TYPE_WRITECOMBINE = 3,
};
static unsigned int cache_type = MEM_TYPE_WRITECOMBINE;
module_param(cache_type, uint, 0444);
MODULE_PARM_DESC(cache_type,
"Memory cache type: 1-cached, 2-uncached, 3-writecombine");
struct buffer {
size_t size;
void *vaddr;
struct sg_table *sg_table;
enum dma_data_direction dma_dir;
struct device* client;
int fd; /* Just for tracking */
};
/*
* dmabuf ops
*/
static void de_noncoherent_kunmap(struct dma_buf *buf, unsigned long page,
void *vaddr);
static void de_noncoherent_release(struct dma_buf *buf)
{
struct buffer *buffer = buf->priv;
struct scatterlist *sgl;
pr_info("%s fd:%d\n", __func__, buffer->fd);
if (unlikely(buffer->vaddr))
de_noncoherent_kunmap(buf, 0, buffer->vaddr);
sgl = buffer->sg_table->sgl;
while (sgl) {
struct page *page = sg_page(sgl);
if (page) {
#ifdef CONFIG_X86
set_memory_wb((unsigned long)page_address(page), 1);
#endif
__free_page(page);
}
sgl = sg_next(sgl);
}
sg_free_table(buffer->sg_table);
kfree(buffer->sg_table);
kfree(buffer);
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,19,0)
static void *de_noncoherent_kmap_atomic(struct dma_buf *buf, unsigned long page)
{
pr_debug("%s\n", __func__);
return NULL;
}
#endif
static struct sg_table *de_noncoherent_map_dma(struct dma_buf_attachment *attach,
enum dma_data_direction dir)
{
struct buffer *buffer = attach->dmabuf->priv;
struct scatterlist *sgl = buffer->sg_table->sgl;
pr_info("%s\n", __func__);
if (buffer->client) {
pr_err("%s client already attached!\n", __func__);
return NULL;
}
/* We are only checking if buffer is mapable */
while (sgl) {
struct page *page = sg_page(sgl);
dma_addr_t dma_addr;
pr_debug("%s:%d phys %#llx length %d\n",
__func__, __LINE__,
(unsigned long long)sg_phys(sgl), sgl->length);
if(!page)
WARN_ONCE(1, "Page does not exist!");
dma_addr = dma_map_page(attach->dev, page, 0, PAGE_SIZE,
DMA_BIDIRECTIONAL);
if (dma_mapping_error(attach->dev, dma_addr)) {
pr_err("%s dma_map_page failed!\n", __func__);
return NULL;;
}
dma_unmap_page(attach->dev, dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL);
#ifdef CONFIG_X86
{
if (cache_type == MEM_TYPE_CACHED)
set_memory_wb((unsigned long)page_address(page), 1);
else if (cache_type == MEM_TYPE_WRITECOMBINE)
set_memory_wc((unsigned long)page_address(page), 1);
else if (cache_type == MEM_TYPE_UNCACHED)
set_memory_uc((unsigned long)page_address(page), 1);
}
#endif
sgl = sg_next(sgl);
}
buffer->client = attach->dev;
return buffer->sg_table;
}
static void de_noncoherent_unmap_dma(struct dma_buf_attachment *attach,
struct sg_table *sgt,
enum dma_data_direction dir)
{
struct buffer *buffer = attach->dmabuf->priv;
pr_info("%s\n", __func__);
buffer->client = NULL;
}
static int de_noncoherent_begin_cpu_access(struct dma_buf *dmabuf,
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
size_t start, size_t len,
#endif
enum dma_data_direction direction)
{
struct buffer *buffer = dmabuf->priv;
struct sg_table *sgt = buffer->sg_table;
int ret;
pr_info("%s\n", __func__);
if (!buffer->client) {
pr_err("%s client is NULL\n", __func__);
return -EFAULT;
}
if (buffer->dma_dir == DMA_NONE) {
ret = dma_map_sg(buffer->client, sgt->sgl, sgt->orig_nents,
direction);
if (ret <= 0) {
pr_err("%s dma_map_sg failed!\n", __func__);
return -EFAULT;
}
sgt->nents = ret;
buffer->dma_dir = direction;
}
if (buffer->dma_dir == DMA_FROM_DEVICE)
dma_sync_sg_for_cpu(buffer->client, sgt->sgl, sgt->orig_nents,
DMA_FROM_DEVICE);
return 0;
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
static void de_noncoherent_end_cpu_access(struct dma_buf *dmabuf,
size_t start, size_t len,
enum dma_data_direction direction)
#else
static int de_noncoherent_end_cpu_access(struct dma_buf *dmabuf,
enum dma_data_direction direction)
#endif
{
struct buffer *buffer = dmabuf->priv;
struct sg_table *sgt = buffer->sg_table;
pr_info("%s\n", __func__);
if (!buffer->client) {
pr_err("%s client is NULL\n", __func__);
goto exit;
}
if (buffer->dma_dir == DMA_NONE)
goto exit;
if (buffer->dma_dir == DMA_TO_DEVICE)
dma_sync_sg_for_cpu(buffer->client, sgt->sgl, sgt->orig_nents,
DMA_TO_DEVICE);
dma_unmap_sg(buffer->client, sgt->sgl,
sgt->orig_nents, buffer->dma_dir);
buffer->dma_dir = DMA_NONE;
exit:
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 6, 0)
return 0;
#endif
;
}
static int de_noncoherent_mmap(struct dma_buf *dmabuf,
struct vm_area_struct *vma)
{
struct buffer *buffer = dmabuf->priv;
struct scatterlist *sgl = buffer->sg_table->sgl;
unsigned long addr;
pr_debug("%s\n", __func__);
/* pgprot_t cached by default */
if (cache_type == MEM_TYPE_WRITECOMBINE)
vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
else if (cache_type == MEM_TYPE_UNCACHED)
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
addr = vma->vm_start;
while (sgl) {
dma_addr_t phys = sg_phys(sgl); /* sg_dma_address ? */
unsigned long pfn = phys >> PAGE_SHIFT;
unsigned int len = sgl->length;
int ret;
ret = remap_pfn_range(vma, addr, pfn, len, vma->vm_page_prot);
if (ret)
return ret;
addr += len;
sgl = sg_next(sgl);
}
return 0;
}
static void *de_noncoherent_kmap(struct dma_buf *dma_buf, unsigned long page)
{
struct buffer *buffer = dma_buf->priv;
struct scatterlist *sgl = buffer->sg_table->sgl;
unsigned int num_pages = sg_nents(sgl);
struct page **pages;
pgprot_t prot;
int i;
pr_debug("%s\n", __func__);
/* NOTE: Ignoring pages param, we have the info info sgt */
if (buffer->vaddr)
return buffer->vaddr;
pages = kmalloc_array(num_pages, sizeof(struct page *), GFP_KERNEL);
if (!pages) {
pr_err("%s failed to allocate memory for pages\n", __func__);
return NULL;
}
prot = PAGE_KERNEL;
/* CACHED by default */
if (cache_type == MEM_TYPE_WRITECOMBINE)
prot = pgprot_writecombine(prot);
else if (cache_type == MEM_TYPE_UNCACHED)
prot = pgprot_noncached(prot);
i = 0;
while (sgl) {
pages[i++] = sg_page(sgl);
sgl = sg_next(sgl);
}
buffer->vaddr = vmap(pages, num_pages, VM_MAP, prot);
kfree(pages);
return buffer->vaddr;
}
static void de_noncoherent_kunmap(struct dma_buf *buf, unsigned long page,
void *vaddr)
{
struct buffer *buffer = buf->priv;
pr_debug("%s\n", __func__);
if (buffer->vaddr != vaddr || !buffer->vaddr) {
pr_warn("%s called with wrong address %p != %p\n",
__func__, vaddr, buffer->vaddr);
return;
}
vunmap(buffer->vaddr);
buffer->vaddr = NULL;
}
static void *de_noncoherent_vmap(struct dma_buf *buf)
{
return de_noncoherent_kmap(buf, 0);
}
static void de_noncoherent_vunmap(struct dma_buf *buf, void *kptr)
{
de_noncoherent_kunmap(buf, 0, kptr);
}
static const struct dma_buf_ops dmabuf_ops = {
.attach = NULL, /* optional */
.detach = NULL, /* optional */
.map_dma_buf = de_noncoherent_map_dma,
.unmap_dma_buf = de_noncoherent_unmap_dma,
.release = de_noncoherent_release,
.begin_cpu_access = de_noncoherent_begin_cpu_access,
.end_cpu_access = de_noncoherent_end_cpu_access,
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,12,0)
.kmap_atomic = de_noncoherent_kmap_atomic,
.kunmap_atomic = NULL, /* optional */
.kmap = de_noncoherent_kmap,
.kunmap = de_noncoherent_kunmap, /* optional */
#else
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,19,0)
.map_atomic = de_noncoherent_kmap_atomic,
.unmap_atomic = NULL, /* optional */
#endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
.map = de_noncoherent_kmap,
.unmap = de_noncoherent_kunmap, /* optional */
#endif
#endif
.mmap = de_noncoherent_mmap,
.vmap = de_noncoherent_vmap,
.vunmap = de_noncoherent_vunmap,
};
int de_heap_buffer_create(size_t size, unsigned long align, void **private_data)
{
struct buffer *buffer;
struct dma_buf *dma_buf;
struct scatterlist *sgl;
int ret;
int pages;
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,1,0)
DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
#endif
pr_info("%s:noncoherent size %zu\n", __func__, size);
buffer = kzalloc(sizeof(struct buffer), GFP_KERNEL);
if (!buffer) {
pr_err("%s:noncoherent failed to allocate buffer\n", __func__);
return -ENOMEM;
}
buffer->size = size;
buffer->sg_table = kzalloc(sizeof(struct sg_table), GFP_KERNEL);
if (!buffer->sg_table) {
pr_err("%s:noncoherent failed to allocate sg_table\n", __func__);
ret = -ENOMEM;
goto sg_table_malloc_failed;
}
pages = (size + PAGE_SIZE - 1) / PAGE_SIZE;
ret = sg_alloc_table(buffer->sg_table, pages, GFP_KERNEL);
if (ret) {
pr_err("%s:noncoherent sg_alloc_table failed\n", __func__);
goto sg_alloc_table_failed;
}
sgl = buffer->sg_table->sgl;
while (sgl) {
struct page *page;
page = alloc_page(MEMORY_ALLOCATION_FLAGS);
if (!page) {
pr_err("%s alloc_page failed!\n", __func__);
ret = -ENOMEM;
goto alloc_page_failed;
}
sg_set_page(sgl, page, PAGE_SIZE, 0);
sgl = sg_next(sgl);
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(3,17,0)
dma_buf = dma_buf_export(buffer, &dmabuf_ops, size, O_RDWR);
#elif LINUX_VERSION_CODE < KERNEL_VERSION(4,1,0)
dma_buf = dma_buf_export(buffer, &dmabuf_ops, size, O_RDWR, NULL);
#else
exp_info.ops = &dmabuf_ops;
exp_info.size = size;
exp_info.flags = O_RDWR;
exp_info.priv = buffer;
exp_info.resv = NULL;
dma_buf = dma_buf_export(&exp_info);
#endif
if (IS_ERR(dma_buf)) {
pr_err("%s:noncoherent dma_buf_export failed\n", __func__);
ret = PTR_ERR(dma_buf);
goto dma_buf_export_failed;
}
buffer->dma_dir = DMA_NONE;
dma_buf->priv = buffer;
*private_data = dma_buf;
pr_info("%s:noncoherent size %zu\n",
__func__, size);
return 0;
alloc_page_failed:
sgl = buffer->sg_table->sgl;
while (sgl) {
struct page *page = sg_page(sgl);
if (page) {
#ifdef CONFIG_X86
set_memory_wb((unsigned long)page_address(page), 1);
#endif
__free_page(page);
}
sgl = sg_next(sgl);
}
dma_buf_export_failed:
sg_free_table(buffer->sg_table);
sg_alloc_table_failed:
kfree(buffer->sg_table);
sg_table_malloc_failed:
kfree(buffer);
return ret;
}
int de_heap_export_fd(void *private_data, unsigned long flags)
{
struct dma_buf *dma_buf = private_data;
struct buffer *buffer = dma_buf->priv;
int ret;
pr_debug("%s:noncoherent %p\n", __func__, dma_buf);
get_dma_buf(dma_buf);
buffer->fd = ret = dma_buf_fd(dma_buf, flags);
if (ret < 0) {
pr_err("%s:noncoherent dma_buf_fd failed\n", __func__);
dma_buf_put(dma_buf);
return ret;
}
pr_info("%s:noncoherent export fd %d\n",
__func__, ret);
return ret;
}
void de_heap_buffer_free(void *private_data)
{
struct dma_buf *dma_buf = private_data;
struct buffer *buffer = dma_buf->priv;
pr_info("%s:noncoherent fd:%d\n", __func__, buffer->fd);
dma_buf_put(dma_buf);
}
int de_heap_heap_init(void)
{
pr_info("%s:noncoherent cache_type:%d\n", __func__, cache_type);
return 0;
}
void de_heap_heap_deinit(void)
{
pr_info("%s:noncoherent\n", __func__);
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

77
drivers/nna/dmabuf_exporter/test/dma-map.c Normal file
View File

@@ -0,0 +1,77 @@
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <errno.h>
#include <errno.h>
#include <string.h>
#include <inttypes.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <dmabuf_exporter.h>
int main(int argc, char **argv)
{
int fd;
int buff_fd;
unsigned long bufSize;
void *uptr;
int i;
if (argc < 2) {
printf("%s buffersize\n", argv[0]);
return 1;
}
bufSize = atoi(argv[1]);
fd = open("/dev/dmabuf", O_RDWR);
if (fd < 0) {
perror("open");
return 1;
}
if (ioctl(fd, DMABUF_IOCTL_CREATE, bufSize)) {
perror("ioctl DMABUF_IOCTL_CREATE");
return 1;
}
buff_fd = ioctl(fd, DMABUF_IOCTL_EXPORT, 0);
if (buff_fd < 0) {
printf("error exporting\n");
return 1;
}
printf("export fd : %d\n", buff_fd);
uptr = mmap(NULL, bufSize, PROT_READ | PROT_WRITE, MAP_SHARED, buff_fd, 0);
if (uptr == MAP_FAILED) {
perror("mmap");
return 1;
}
printf("mapped to %p\n", uptr);
for (i = 0; getpagesize() * i < bufSize; i++) {
int *iptr = (int *)((uintptr_t)uptr + getpagesize() * i);
int val = (0xbeef << 16) | i;
printf("write [%p] = %x\n", iptr, val);
*iptr = val;
}
for (i = 0; getpagesize() * i < bufSize; i++) {
int *iptr = (int *)((uintptr_t)uptr + getpagesize() * i);
int val = (0xbeef << 16) | i;
printf("read [%p] = %x [expected %x]\n", iptr, *iptr, val);
if (*iptr != val) {
printf("!!! ERROR !!!\n");
return 1;
}
}
sleep(3);
close(buff_fd);
close(fd);
return 0;
}

61
drivers/nna/dmabuf_exporter/test/dma-test.c Normal file
View File

@@ -0,0 +1,61 @@
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <errno.h>
#include <errno.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <dmabuf_exporter.h>
int main(int argc, char **argv)
{
int fd;
int buff_fd;
int nBuffs;
unsigned long bufSize;
int i;
if (argc < 3) {
printf("%s <# buffers> buffersize\n", argv[0]);
return 1;
}
nBuffs = atoi(argv[1]);
bufSize = atoi(argv[2]);
for (i = 0; i < nBuffs; i++) {
void *uptr;
fd = open("/dev/dmabuf", O_RDWR);
if (fd < 0) {
perror("open");
return 1;
}
if (ioctl(fd, DMABUF_IOCTL_CREATE, bufSize)) {
perror("ioctl DMABUF_IOCTL_CREATE");
return 1;
}
buff_fd = ioctl(fd, DMABUF_IOCTL_EXPORT, 0);
if (buff_fd < 0) {
printf("error exporting\n");
return 1;
}
printf("export fd : %d\n", buff_fd);
uptr = mmap(NULL, bufSize, PROT_READ | PROT_WRITE, MAP_SHARED, buff_fd, 0);
if (uptr == MAP_FAILED) {
perror("mmap");
return 1;
}
printf("mapped to %p\n", uptr);
}
sleep(5);
return 0;
}

20
drivers/nna/dmabuf_exporter/uapi/dmabuf_exporter.h Normal file
View File

@@ -0,0 +1,20 @@
/*
User API for dmabuf_exporter
*/
#ifndef _DMABUF_EXPORTER_H
#define _DMABUF_EXPORTER_H
#include <linux/ioctl.h>
#include <linux/types.h>
#define DMABUF_IOCTL_BASE 'D'
#define DMABUF_IOCTL_CREATE _IOR(DMABUF_IOCTL_BASE, 0, unsigned long)
#define DMABUF_IOCTL_EXPORT _IOR(DMABUF_IOCTL_BASE, 1, unsigned long)
#ifdef CONFIG_COMPAT
#define COMPAT_DMABUF_IOCTL_CREATE _IOR(DMABUF_IOCTL_BASE, 0, unsigned int)
#define COMPAT_DMABUF_IOCTL_EXPORT _IOR(DMABUF_IOCTL_BASE, 1, unsigned int)
#endif
#endif /* _DMABUF_EXPORTER_H */

136
drivers/nna/dmabuf_exporter/uapi/kernel_4x14/ion.h Normal file
View File

@@ -0,0 +1,136 @@
/*
* drivers/staging/android/uapi/ion.h
*
* Copyright (C) 2011 Google, Inc.
*
* This software is licensed under the terms of the GNU General Public
* License version 2, as published by the Free Software Foundation, and
* may be copied, distributed, and modified under those terms.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
*/
#ifndef _UAPI_LINUX_ION_H
#define _UAPI_LINUX_ION_H
#include <linux/ioctl.h>
#include <linux/types.h>
/**
* enum ion_heap_types - list of all possible types of heaps
* @ION_HEAP_TYPE_SYSTEM: memory allocated via vmalloc
* @ION_HEAP_TYPE_SYSTEM_CONTIG: memory allocated via kmalloc
* @ION_HEAP_TYPE_CARVEOUT: memory allocated from a prereserved
* carveout heap, allocations are physically
* contiguous
* @ION_HEAP_TYPE_DMA: memory allocated via DMA API
* @ION_NUM_HEAPS: helper for iterating over heaps, a bit mask
* is used to identify the heaps, so only 32
* total heap types are supported
*/
enum ion_heap_type {
ION_HEAP_TYPE_SYSTEM,
ION_HEAP_TYPE_SYSTEM_CONTIG,
ION_HEAP_TYPE_CARVEOUT,
ION_HEAP_TYPE_CHUNK,
ION_HEAP_TYPE_DMA,
ION_HEAP_TYPE_CUSTOM, /*
* must be last so device specific heaps always
* are at the end of this enum
*/
};
#define ION_NUM_HEAP_IDS (sizeof(unsigned int) * 8)
/**
* allocation flags - the lower 16 bits are used by core ion, the upper 16
* bits are reserved for use by the heaps themselves.
*/
/*
* mappings of this buffer should be cached, ion will do cache maintenance
* when the buffer is mapped for dma
*/
#define ION_FLAG_CACHED 1
/**
* DOC: Ion Userspace API
*
* create a client by opening /dev/ion
* most operations handled via following ioctls
*
*/
/**
* struct ion_allocation_data - metadata passed from userspace for allocations
* @len: size of the allocation
* @heap_id_mask: mask of heap ids to allocate from
* @flags: flags passed to heap
* @handle: pointer that will be populated with a cookie to use to
* refer to this allocation
*
* Provided by userspace as an argument to the ioctl
*/
struct ion_allocation_data {
__u64 len;
__u32 heap_id_mask;
__u32 flags;
__u32 fd;
__u32 unused;
};
#define MAX_HEAP_NAME 32
/**
* struct ion_heap_data - data about a heap
* @name - first 32 characters of the heap name
* @type - heap type
* @heap_id - heap id for the heap
*/
struct ion_heap_data {
char name[MAX_HEAP_NAME];
__u32 type;
__u32 heap_id;
__u32 reserved0;
__u32 reserved1;
__u32 reserved2;
};
/**
* struct ion_heap_query - collection of data about all heaps
* @cnt - total number of heaps to be copied
* @heaps - buffer to copy heap data
*/
struct ion_heap_query {
__u32 cnt; /* Total number of heaps to be copied */
__u32 reserved0; /* align to 64bits */
__u64 heaps; /* buffer to be populated */
__u32 reserved1;
__u32 reserved2;
};
#define ION_IOC_MAGIC 'I'
/**
* DOC: ION_IOC_ALLOC - allocate memory
*
* Takes an ion_allocation_data struct and returns it with the handle field
* populated with the opaque handle for the allocation.
*/
#define ION_IOC_ALLOC _IOWR(ION_IOC_MAGIC, 0, \
struct ion_allocation_data)
/**
* DOC: ION_IOC_HEAP_QUERY - information about available heaps
*
* Takes an ion_heap_query structure and populates information about
* available Ion heaps.
*/
#define ION_IOC_HEAP_QUERY _IOWR(ION_IOC_MAGIC, 8, \
struct ion_heap_query)
#endif /* _UAPI_LINUX_ION_H */

203
drivers/nna/dmabuf_exporter/uapi/kernel_4x4/ion.h Normal file
View File

@@ -0,0 +1,203 @@
/*
* drivers/staging/android/uapi/ion.h
*
* Copyright (C) 2011 Google, Inc.
*
* This software is licensed under the terms of the GNU General Public
* License version 2, as published by the Free Software Foundation, and
* may be copied, distributed, and modified under those terms.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
*/
#ifndef _UAPI_LINUX_ION_H
#define _UAPI_LINUX_ION_H
#include <linux/ioctl.h>
#include <linux/types.h>
typedef int ion_user_handle_t;
/**
* enum ion_heap_types - list of all possible types of heaps
* @ION_HEAP_TYPE_SYSTEM: memory allocated via vmalloc
* @ION_HEAP_TYPE_SYSTEM_CONTIG: memory allocated via kmalloc
* @ION_HEAP_TYPE_CARVEOUT: memory allocated from a prereserved
* carveout heap, allocations are physically
* contiguous
* @ION_HEAP_TYPE_DMA: memory allocated via DMA API
* @ION_NUM_HEAPS: helper for iterating over heaps, a bit mask
* is used to identify the heaps, so only 32
* total heap types are supported
*/
enum ion_heap_type {
ION_HEAP_TYPE_SYSTEM,
ION_HEAP_TYPE_SYSTEM_CONTIG,
ION_HEAP_TYPE_CARVEOUT,
ION_HEAP_TYPE_CHUNK,
ION_HEAP_TYPE_DMA,
ION_HEAP_TYPE_CUSTOM, /*
* must be last so device specific heaps always
* are at the end of this enum
*/
ION_NUM_HEAPS = 16,
};
#define ION_HEAP_SYSTEM_MASK (1 << ION_HEAP_TYPE_SYSTEM)
#define ION_HEAP_SYSTEM_CONTIG_MASK (1 << ION_HEAP_TYPE_SYSTEM_CONTIG)
#define ION_HEAP_CARVEOUT_MASK (1 << ION_HEAP_TYPE_CARVEOUT)
#define ION_HEAP_TYPE_DMA_MASK (1 << ION_HEAP_TYPE_DMA)
#define ION_NUM_HEAP_IDS (sizeof(unsigned int) * 8)
/**
* allocation flags - the lower 16 bits are used by core ion, the upper 16
* bits are reserved for use by the heaps themselves.
*/
#define ION_FLAG_CACHED 1 /*
* mappings of this buffer should be
* cached, ion will do cache
* maintenance when the buffer is
* mapped for dma
*/
#define ION_FLAG_CACHED_NEEDS_SYNC 2 /*
* mappings of this buffer will created
* at mmap time, if this is set
* caches must be managed
* manually
*/
/**
* DOC: Ion Userspace API
*
* create a client by opening /dev/ion
* most operations handled via following ioctls
*
*/
/**
* struct ion_allocation_data - metadata passed from userspace for allocations
* @len: size of the allocation
* @align: required alignment of the allocation
* @heap_id_mask: mask of heap ids to allocate from
* @flags: flags passed to heap
* @handle: pointer that will be populated with a cookie to use to
* refer to this allocation
*
* Provided by userspace as an argument to the ioctl
*/
struct ion_allocation_data {
size_t len;
size_t align;
unsigned int heap_id_mask;
unsigned int flags;
ion_user_handle_t handle;
};
/**
* struct ion_fd_data - metadata passed to/from userspace for a handle/fd pair
* @handle: a handle
* @fd: a file descriptor representing that handle
*
* For ION_IOC_SHARE or ION_IOC_MAP userspace populates the handle field with
* the handle returned from ion alloc, and the kernel returns the file
* descriptor to share or map in the fd field. For ION_IOC_IMPORT, userspace
* provides the file descriptor and the kernel returns the handle.
*/
struct ion_fd_data {
ion_user_handle_t handle;
int fd;
};
/**
* struct ion_handle_data - a handle passed to/from the kernel
* @handle: a handle
*/
struct ion_handle_data {
ion_user_handle_t handle;
};
/**
* struct ion_custom_data - metadata passed to/from userspace for a custom ioctl
* @cmd: the custom ioctl function to call
* @arg: additional data to pass to the custom ioctl, typically a user
* pointer to a predefined structure
*
* This works just like the regular cmd and arg fields of an ioctl.
*/
struct ion_custom_data {
unsigned int cmd;
unsigned long arg;
};
#define ION_IOC_MAGIC 'I'
/**
* DOC: ION_IOC_ALLOC - allocate memory
*
* Takes an ion_allocation_data struct and returns it with the handle field
* populated with the opaque handle for the allocation.
*/
#define ION_IOC_ALLOC _IOWR(ION_IOC_MAGIC, 0, \
struct ion_allocation_data)
/**
* DOC: ION_IOC_FREE - free memory
*
* Takes an ion_handle_data struct and frees the handle.
*/
#define ION_IOC_FREE _IOWR(ION_IOC_MAGIC, 1, struct ion_handle_data)
/**
* DOC: ION_IOC_MAP - get a file descriptor to mmap
*
* Takes an ion_fd_data struct with the handle field populated with a valid
* opaque handle. Returns the struct with the fd field set to a file
* descriptor open in the current address space. This file descriptor
* can then be used as an argument to mmap.
*/
#define ION_IOC_MAP _IOWR(ION_IOC_MAGIC, 2, struct ion_fd_data)
/**
* DOC: ION_IOC_SHARE - creates a file descriptor to use to share an allocation
*
* Takes an ion_fd_data struct with the handle field populated with a valid
* opaque handle. Returns the struct with the fd field set to a file
* descriptor open in the current address space. This file descriptor
* can then be passed to another process. The corresponding opaque handle can
* be retrieved via ION_IOC_IMPORT.
*/
#define ION_IOC_SHARE _IOWR(ION_IOC_MAGIC, 4, struct ion_fd_data)
/**
* DOC: ION_IOC_IMPORT - imports a shared file descriptor
*
* Takes an ion_fd_data struct with the fd field populated with a valid file
* descriptor obtained from ION_IOC_SHARE and returns the struct with the handle
* filed set to the corresponding opaque handle.
*/
#define ION_IOC_IMPORT _IOWR(ION_IOC_MAGIC, 5, struct ion_fd_data)
/**
* DOC: ION_IOC_SYNC - syncs a shared file descriptors to memory
*
* Deprecated in favor of using the dma_buf api's correctly (syncing
* will happen automatically when the buffer is mapped to a device).
* If necessary should be used after touching a cached buffer from the cpu,
* this will make the buffer in memory coherent.
*/
#define ION_IOC_SYNC _IOWR(ION_IOC_MAGIC, 7, struct ion_fd_data)
/**
* DOC: ION_IOC_CUSTOM - call architecture specific ion ioctl
*
* Takes the argument of the architecture specific ioctl to call and
* passes appropriate userdata for that ioctl
*/
#define ION_IOC_CUSTOM _IOWR(ION_IOC_MAGIC, 6, struct ion_custom_data)
#endif /* _UAPI_LINUX_ION_H */

3
drivers/nna/fenrir_loki/Makefile Normal file
View File

@@ -0,0 +1,3 @@
obj-${CONFIG_LOKI} += loki.o
loki-y += loki-main.o
loki-y += loki-intc.o

159
drivers/nna/fenrir_loki/loki-intc.c Normal file
View File

@@ -0,0 +1,159 @@
// SPDX-License-Identifier: GPL-2.0+
/*
* Code for Fenrir's Loki interrupt controller.
*/
#include <linux/delay.h>
#include <linux/irq.h>
#include <linux/bitops.h>
#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/interrupt.h>
#include <linux/irqdomain.h>
#include <linux/irqchip.h>
#include <linux/of.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/of_platform.h>
#include <linux/irqchip.h>
#include <linux/irqdomain.h>
#include "loki.h"
static void loki_mask_irq(struct irq_data *d)
{
struct loki_drvdata *pdata = irq_data_get_irq_chip_data(d);
u32 reg = pdata->readreg32(pdata, REG_LOKI_INTERRUPT_ENABLE);
pdata->writereg32(pdata, REG_LOKI_INTERRUPT_ENABLE, reg & ~(LOKI_INTERRUPT_DUT0));
}
static void loki_unmask_irq(struct irq_data *d)
{
struct loki_drvdata *pdata = irq_data_get_irq_chip_data(d);
u32 reg = pdata->readreg32(pdata, REG_LOKI_INTERRUPT_ENABLE);
pdata->writereg32(pdata, REG_LOKI_INTERRUPT_ENABLE, reg | LOKI_INTERRUPT_DUT0);
}
static struct irq_chip fenrir_loki = {
.name = "loki-intc",
.irq_mask = loki_mask_irq,
.irq_unmask = loki_unmask_irq,
};
static int loki_intc_map(struct irq_domain *d, unsigned int irq, irq_hw_number_t hw)
{
struct irq_chip *chip = &fenrir_loki;
irq_domain_set_info(d, irq, hw, chip, d->host_data,
handle_level_irq, NULL, NULL);
irq_set_status_flags(irq, IRQ_LEVEL);
return 0;
}
static irqreturn_t loki_isrcb(int irq, void *dev_id)
{
struct platform_device *pdev = (struct platform_device *)dev_id;
struct loki_drvdata *pdata;
u32 reg, mask, timeout;
if (!pdev) {
pr_err("LOKI: pdev not set!?\n");
return IRQ_NONE;
}
pdata = platform_get_drvdata(pdev);
if (!pdata) {
pr_err("LOKI: pdata not set!?\n");
return IRQ_NONE;
}
reg = pdata->readreg32(pdata, REG_LOKI_INTERRUPT_STATUS);
mask = pdata->readreg32(pdata, REG_LOKI_INTERRUPT_ENABLE);
timeout = pdata->readreg32(pdata, REG_LOKI_INTERRUPT_TIMEOUT_CLR);
dev_dbg(&pdev->dev, "Got an interrupt. %X - %X - %X\n", reg, mask, timeout);
/* Check the timeout register just in case */
if (timeout != 0) {
dev_warn(&pdev->dev, "Interrupt timeout fired. Will need to be reset\n");
}
if (reg & mask) {
if (reg & LOKI_INTERRUPT_TESTINT) {
dev_warn(&pdev->dev, "Test interrupt fired! Was it on purpose?\n");
/* Disable the interrupt */
pdata->writereg32(pdata, REG_LOKI_INTERRUPT_TEST, 0);
}
else if (reg & LOKI_INTERRUPT_DUT0) {
int logical_irq_num;
/* trigger registered IRQ, if any */
logical_irq_num = irq_find_mapping(pdata->intc.domain, 0);
generic_handle_irq(logical_irq_num);
}
/* Clear interrupts */
pdata->writereg32(pdata, REG_LOKI_INTERRUPT_CLR, reg);
return IRQ_HANDLED;
}
return IRQ_NONE;
}
static const struct irq_domain_ops loki_intc_ops = {
.xlate = irq_domain_xlate_onecell,
.map = loki_intc_map,
};
int loki_intc_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct device_node *of_node = pdev->dev.of_node;
int ret = 0;
struct loki_drvdata *priv_data = platform_get_drvdata(pdev);;
dev_dbg(dev, "Going to register loki's intc...");
/* Setup the interrupt controller */
priv_data->writereg32(priv_data, REG_LOKI_INTERRUPT_ENABLE, LOKI_INTERRUPT_BASE);
priv_data->writereg32(priv_data, REG_LOKI_INTERRUPT_CLR, LOKI_INTERRUPT_BASE | LOKI_INTERRUPT_DUT0);
priv_data->writereg32(priv_data, REG_LOKI_INTERRUPT_TIMEOUT_CLR, 0x2);
priv_data->intc.irq_num = irq_of_parse_and_map(of_node, 0);
if (priv_data->intc.irq_num == 0) {
dev_err(dev, "Could not map IRQ\n");
ret = -ENXIO;
goto exit;
}
ret = devm_request_irq(dev, priv_data->intc.irq_num, &loki_isrcb, IRQF_SHARED, DEVICE_NAME, pdev);
if (ret) {
dev_err(dev, "Failed to request irq\n");
ret = -ENXIO;
goto exit;
}
priv_data->intc.domain = irq_domain_add_linear(of_node, 1, &loki_intc_ops, priv_data);
if (!priv_data->intc.domain) {
dev_err(dev, "Unable to create IRQ domain\n");
ret = -ENXIO;
goto exit;
}
exit:
return ret;
}
int loki_intc_remove(struct platform_device *pdev)
{
struct loki_drvdata *pdata = platform_get_drvdata(pdev);
irq_dispose_mapping(irq_find_mapping(pdata->intc.domain, 0));
irq_domain_remove(pdata->intc.domain);
return 0;
}

110
drivers/nna/fenrir_loki/loki-main.c Normal file
View File

@@ -0,0 +1,110 @@
// SPDX-License-Identifier: GPL-2.0+
/*
* Code for Fenrir's Loki.
*/
#include <linux/delay.h>
#include <linux/irq.h>
#include <linux/bitops.h>
#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/of.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/of_platform.h>
#include "loki.h"
static inline unsigned int loki_readreg32(struct loki_drvdata *pdata, unsigned long offset)
{
void __iomem *reg = (void __iomem *)pdata->regbase + offset;
return ioread32(reg);
}
static inline void loki_writereg32(struct loki_drvdata *pdata, unsigned long offset, int val)
{
void __iomem *reg = (void __iomem *)pdata->regbase + offset;
iowrite32(val, reg);
}
static int loki_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct device_node *of_node = pdev->dev.of_node;
int ret = 0;
struct loki_drvdata *priv_data;
uint32_t memif_cache, memif_prot;
dev_dbg(dev, "Probe...");
priv_data = devm_kzalloc(dev, sizeof(struct loki_drvdata), GFP_KERNEL);
if (!priv_data) {
pr_err("Memory allocation error, aborting.\n");
ret = -ENOMEM;
goto exit;
}
priv_data->regbase = of_iomap(of_node, 0);
if (!priv_data->regbase) {
dev_err(dev, "Unable to map local interrupt registers\n");
ret = -ENXIO;
goto exit;
}
priv_data->writereg32 = loki_writereg32;
priv_data->readreg32 = loki_readreg32;
/* Reset the DUT */
priv_data->writereg32(priv_data, REG_LOKI_EXTERNAL_RESET, 0);
udelay(10);
priv_data->writereg32(priv_data, REG_LOKI_EXTERNAL_RESET, 1);
platform_set_drvdata(pdev, priv_data);
/* Get optional data from the Device Tree */
if (!of_property_read_u64(pdev->dev.of_node, "memif-cache",
(uint64_t *)&memif_cache)) {
dev_info(dev, "Setting memif_cache to %X from the DT\n", memif_cache);
}
priv_data->writereg32(priv_data, REG_LOKI_MEMIF_CACHE_SET, memif_cache);
if (!of_property_read_u64(pdev->dev.of_node, "memif-prot",
(uint64_t *)&memif_prot)) {
dev_info(dev, "Setting memif_prot to %X from the DT\n", memif_prot);
}
priv_data->writereg32(priv_data, REG_LOKI_MEMIF_PROT_SET, memif_prot);
loki_intc_probe(pdev);
exit:
return ret;
}
static int loki_remove(struct platform_device *pdev)
{
struct loki_drvdata *pdata = platform_get_drvdata(pdev);
loki_intc_remove(pdev);
return 0;
}
static const struct of_device_id loki_dt_ids[] = {
{ .compatible = "img,loki", },
{},
};
MODULE_DEVICE_TABLE(of, loki_dt_ids);
static struct platform_driver loki_device_driver = {
.probe = loki_probe,
.remove = loki_remove,
.driver = {
.name = DEVICE_NAME,
.of_match_table = of_match_ptr(loki_dt_ids),
}
};
module_platform_driver(loki_device_driver);
MODULE_AUTHOR("Imagination Technologies");
MODULE_DESCRIPTION("Fenrir Loki driver");
MODULE_LICENSE("GPL v2");

44
drivers/nna/fenrir_loki/loki.h Normal file
View File

@@ -0,0 +1,44 @@
// SPDX-License-Identifier: GPL-2.0+
/*
* Fenrir's Loki header
*/
#ifndef LOKI_H
#define LOKI_H
#define DEVICE_NAME "loki_intc"
#define REG_LOKI_EXTERNAL_RESET (0x0084)
#define REG_LOKI_INTERRUPT_STATUS (0x0100)
#define REG_LOKI_INTERRUPT_ENABLE (0x0104)
#define REG_LOKI_INTERRUPT_CLR (0x010C)
#define REG_LOKI_INTERRUPT_TEST (0x0110)
#define REG_LOKI_INTERRUPT_TIMEOUT_CLR (0x0114)
#define REG_LOKI_INTERRUPT_TIMEOUT (0x0118)
#define REG_LOKI_MEMIF_CACHE_SET (0x0230)
#define REG_LOKI_MEMIF_PROT_SET (0x0234)
/* interrupt bits definitions */
#define LOKI_INTERRUPT_MASTER_ENABLE (1 << 31)
#define LOKI_INTERRUPT_TESTINT (1 << 30)
#define LOKI_INTERRUPT_DUT0 (1 << 0)
#define LOKI_INTERRUPT_BASE (LOKI_INTERRUPT_MASTER_ENABLE | LOKI_INTERRUPT_TESTINT)
struct loki_intc_drvdata {
struct irq_domain *domain;
int irq_num;
};
struct loki_drvdata {
void __iomem *regbase;
unsigned int (*readreg32)(struct loki_drvdata *pdata, unsigned long offset);
void (*writereg32)(struct loki_drvdata *pdata, unsigned long offset, int val);
struct loki_intc_drvdata intc;
};
int loki_intc_probe(struct platform_device *pdev);
int loki_intc_remove(struct platform_device *pdev);
#endif /* LOKI_H */

33
drivers/nna/img_mem/Makefile Normal file
View File

@@ -0,0 +1,33 @@
img_mem-y := img_mem_man.o
img_mem-$(CONFIG_GENERIC_ALLOCATOR) += img_mem_carveout.o
# removed building ION, kernel 4.14 ABI changes not implemented yet
img_mem-$(CONFIG_ION) += img_mem_ion.o
img_mem-$(CONFIG_DMA_SHARED_BUFFER) += img_mem_dmabuf.o
img_mem-y += img_mem_unified.o img_mem_coherent.o
img_mem-y += img_mem_anonymous.o
img_mem-y += img_mem_ocm.o
img_mem-y += img_pdump.o
ifeq ($(CONFIG_ION), y)
CFLAGS_img_mem_ion.o += -Idrivers/staging/android/ion
endif
# IMGMMU: These should be obsoleted
ccflags-y += -I$(src)/imgmmu/mmulib
# IMGMMU: code
img_mem-y += imgmmu/imgmmu.o
img_mem-y += imgmmu/kernel_heap.o
obj-$(CONFIG_VHA) += img_mem.o
# Alias for backward compatibility
CONFIG_HW_AX3_MC := $(CONFIG_HW_MULTICORE)
# Magna does not use multiple OSes approach
ifeq ($(CONFIG_HW_AX3), y)
ifeq ($(CONFIG_HW_AX3_MC),)
ccflags-y += -DOSID=$(CONFIG_TARGET_OSID)
endif
endif
ccflags-y +=-DDEFAULT_SYMBOL_NAMESPACE=IMG_MEM

395
drivers/nna/img_mem/img_mem_anonymous.c Normal file
View File

@@ -0,0 +1,395 @@
/*!
*****************************************************************************
*
* @File img_mem_anonymous.c
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/scatterlist.h>
#include <linux/gfp.h>
#include <linux/device.h>
#include <linux/vmalloc.h>
#include <linux/sched.h>
#include <linux/dma-mapping.h>
#include <linux/version.h>
#include <img_mem_man.h>
#include "img_mem_man_priv.h"
static int trace_physical_pages;
struct buffer_data {
struct sg_table *sgt;
enum img_mem_attr mattr; /* memory attributes */
};
static int anonymous_heap_import(struct device *device, struct heap *heap,
size_t size, enum img_mem_attr attr, uint64_t buf_hnd,
struct buffer *buffer)
{
struct buffer_data *data;
unsigned long cpu_addr = (unsigned long)buf_hnd;
struct sg_table *sgt;
struct page **pages;
struct scatterlist *sgl;
int num_pages = (size + PAGE_SIZE - 1) / PAGE_SIZE;
int ret;
int i;
pr_debug("%s:%d buffer %d (0x%p) cpu_addr %#lx for PID:%d\n",
__func__, __LINE__, buffer->id, buffer,
cpu_addr, task_pid_nr(current));
/* Check alignment */
if (cpu_addr & (PAGE_SIZE-1)) {
pr_err("%s wrong alignment of %#lx address!\n",
__func__, cpu_addr);
return -EFAULT;
}
pages = kmalloc_array(num_pages, sizeof(struct page *),
GFP_KERNEL | __GFP_ZERO);
if (!pages) {
pr_err("%s failed to allocate memory for pages\n", __func__);
return -ENOMEM;
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 8, 0)
down_read(&current->mm->mmap_sem);
#else
down_read(&current->mm->mmap_lock);
#endif
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 9, 0)
ret = get_user_pages(
cpu_addr, num_pages,
FOLL_WRITE,
pages, NULL);
#else
pr_err("%s get_user_pages not supported for this kernel version\n",
__func__);
ret = -1;
#endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 8, 0)
up_read(&current->mm->mmap_sem);
#else
up_read(&current->mm->mmap_lock);
#endif
if (ret != num_pages) {
pr_err("%s failed to get_user_pages count:%d for %#lx address\n",
__func__, num_pages, cpu_addr);
ret = -ENOMEM;
goto get_user_pages_failed;
}
sgt = kmalloc(sizeof(struct sg_table), GFP_KERNEL);
if (!sgt) {
ret = -ENOMEM;
goto alloc_sgt_failed;
}
ret = sg_alloc_table(sgt, num_pages, GFP_KERNEL);
if (ret) {
pr_err("%s failed to allocate sgt with num_pages\n", __func__);
goto alloc_sgt_pages_failed;
}
data = kmalloc(sizeof(struct buffer_data), GFP_KERNEL);
if (!data) {
ret = -ENOMEM;
goto alloc_priv_failed;
}
for_each_sg(sgt->sgl, sgl, sgt->nents, i) {
struct page *page = pages[i];
sg_set_page(sgl, page, PAGE_SIZE, 0);
/* Sanity check if physical address is
* accessible from the device PoV */
if (~dma_get_mask(device) & sg_phys(sgl)) {
pr_err("%s physical address is out of dma_mask,"
" and probably won't be accessible by the core!\n",
__func__);
ret = -ERANGE;
goto dma_mask_check_failed;
}
if (trace_physical_pages)
pr_info("%s:%d phys %#llx length %d\n",
__func__, __LINE__,
(unsigned long long)sg_phys(sgl), sgl->length);
}
pr_debug("%s:%d buffer %d orig_nents %d\n", __func__, __LINE__,
buffer->id, sgt->orig_nents);
data->sgt = sgt;
data->mattr = attr;
buffer->priv = data;
ret = dma_map_sg(buffer->device, sgt->sgl, sgt->orig_nents,
DMA_BIDIRECTIONAL);
if (ret <= 0) {
pr_err("%s dma_map_sg failed!\n", __func__);
goto dma_mask_check_failed;
}
kfree(pages);
return 0;
dma_mask_check_failed:
kfree(data);
alloc_priv_failed:
sg_free_table(sgt);
alloc_sgt_pages_failed:
kfree(sgt);
get_user_pages_failed:
for (i = 0; i < num_pages; i++)
if (pages[i])
put_page(pages[i]);
alloc_sgt_failed:
kfree(pages);
return ret;
}
static void anonymous_heap_free(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *data = buffer->priv;
struct sg_table *sgt = data->sgt;
struct scatterlist *sgl;
bool dirty = false;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
dma_unmap_sg(buffer->device, sgt->sgl, sgt->orig_nents,
DMA_BIDIRECTIONAL);
if (buffer->kptr) {
pr_debug("%s vunmap 0x%p\n", __func__, buffer->kptr);
dirty = true;
vunmap(buffer->kptr);
buffer->kptr = NULL;
}
sgl = sgt->sgl;
while (sgl) {
struct page *page = sg_page(sgl);
if (page) {
if (dirty)
set_page_dirty(page);
put_page(page);
}
sgl = sg_next(sgl);
}
sg_free_table(sgt);
kfree(sgt);
kfree(data);
}
static int anonymous_heap_map_km(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *buffer_data = buffer->priv;
struct sg_table *sgt = buffer_data->sgt;
struct scatterlist *sgl = sgt->sgl;
unsigned int num_pages = sg_nents(sgl);
struct page **pages;
pgprot_t prot;
int i;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (buffer->kptr) {
pr_warn("%s called for already mapped buffer %d\n",
__func__, buffer->id);
return 0;
}
pages = kmalloc_array(num_pages, sizeof(struct page *), GFP_KERNEL);
if (!pages) {
pr_err("%s failed to allocate memory for pages\n", __func__);
return -ENOMEM;
}
prot = PAGE_KERNEL;
/* CACHED by default */
if (buffer_data->mattr & IMG_MEM_ATTR_WRITECOMBINE)
prot = pgprot_writecombine(prot);
else if (buffer_data->mattr & IMG_MEM_ATTR_UNCACHED)
prot = pgprot_noncached(prot);
i = 0;
while (sgl) {
pages[i++] = sg_page(sgl);
sgl = sg_next(sgl);
}
buffer->kptr = vmap(pages, num_pages, VM_MAP, prot);
kfree(pages);
if (!buffer->kptr) {
pr_err("%s vmap failed!\n", __func__);
return -EFAULT;
}
pr_debug("%s:%d buffer %d vmap to 0x%p\n", __func__, __LINE__,
buffer->id, buffer->kptr);
return 0;
}
static int anonymous_heap_unmap_km(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *data = buffer->priv;
struct sg_table *sgt = data->sgt;
struct scatterlist *sgl;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (!buffer->kptr) {
pr_warn("%s called for unmapped buffer %d\n",
__func__, buffer->id);
return 0;
}
pr_debug("%s:%d buffer %d kunmap from 0x%p\n", __func__, __LINE__,
buffer->id, buffer->kptr);
vunmap(buffer->kptr);
buffer->kptr = NULL;
sgl = sgt->sgl;
while (sgl) {
struct page *page = sg_page(sgl);
if (page) {
set_page_dirty(page);
}
sgl = sg_next(sgl);
}
return 0;
}
static int anonymous_get_sg_table(struct heap *heap, struct buffer *buffer,
struct sg_table **sg_table, bool *use_sg_dma)
{
struct buffer_data *data = buffer->priv;
*sg_table = data->sgt;
*use_sg_dma = false;
return 0;
}
static void anonymous_sync_cpu_to_dev(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *buffer_data = buffer->priv;
struct sg_table *sgt = buffer_data->sgt;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (!(buffer_data->mattr & IMG_MEM_ATTR_UNCACHED)) {
dma_sync_sg_for_device(buffer->device,
sgt->sgl,
sgt->orig_nents,
DMA_TO_DEVICE);
dma_sync_sg_for_cpu(buffer->device,
sgt->sgl,
sgt->orig_nents,
DMA_FROM_DEVICE);
}
}
static void anonymous_sync_dev_to_cpu(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *buffer_data = buffer->priv;
struct sg_table *sgt = buffer_data->sgt;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (!(buffer_data->mattr & IMG_MEM_ATTR_UNCACHED)) {
dma_sync_sg_for_cpu(buffer->device,
sgt->sgl,
sgt->orig_nents,
DMA_FROM_DEVICE);
}
}
static void anonymous_heap_destroy(struct heap *heap)
{
pr_debug("%s:%d\n", __func__, __LINE__);
}
static struct heap_ops anonymous_heap_ops = {
.alloc = NULL,
.import = anonymous_heap_import,
.free = anonymous_heap_free,
.map_um = NULL,
.unmap_um = NULL,
.map_km = anonymous_heap_map_km,
.unmap_km = anonymous_heap_unmap_km,
.get_sg_table = anonymous_get_sg_table,
.get_page_array = NULL,
.sync_cpu_to_dev = anonymous_sync_cpu_to_dev,
.sync_dev_to_cpu = anonymous_sync_dev_to_cpu,
.set_offset = NULL,
.destroy = anonymous_heap_destroy,
};
int img_mem_anonymous_init(const struct heap_config *heap_cfg, struct heap *heap)
{
pr_debug("%s:%d\n", __func__, __LINE__);
heap->ops = &anonymous_heap_ops;
return 0;
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

798
drivers/nna/img_mem/img_mem_carveout.c Normal file
View File

@@ -0,0 +1,798 @@
/*!
*****************************************************************************
*
* @File img_mem_carveout.c
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/scatterlist.h>
#include <linux/dma-buf.h>
#include <linux/gfp.h>
#include <linux/vmalloc.h>
#include <linux/genalloc.h>
#include <linux/version.h>
#include <asm/cacheflush.h>
#include <img_mem_man.h>
#include "img_mem_man_priv.h"
/* Default allocation order */
#define POOL_ALLOC_ORDER_BASE PAGE_SHIFT
struct heap_data {
struct gen_pool *pool;
};
struct buffer_data {
unsigned long addr; /* addr returned by genalloc */
uint64_t *addrs; /* array of physical addresses, upcast to 64-bit */
enum img_mem_attr mattr; /* memory attributes */
struct vm_area_struct *mapped_vma; /* Needed for cache manipulation */
/* exporter via dmabuf */
struct sg_table *sgt;
bool exported;
struct dma_buf *dma_buf;
dma_addr_t dma_base;
unsigned int dma_size;
};
static int trace_physical_pages;
static int trace_mmap_fault;
/*
* dmabuf wrapper ops
*/
static struct sg_table *carveout_map_dmabuf(struct dma_buf_attachment *attach,
enum dma_data_direction dir)
{
struct buffer *buffer = attach->dmabuf->priv;
struct buffer_data *buffer_data;
if (!buffer)
return NULL;
pr_debug("%s\n", __func__);
buffer_data = buffer->priv;
sg_dma_address(buffer_data->sgt->sgl) = buffer_data->dma_base;
sg_dma_len(buffer_data->sgt->sgl) = buffer_data->dma_size;
return buffer_data->sgt;
}
static void carveout_unmap_dmabuf(struct dma_buf_attachment *attach,
struct sg_table *sgt,
enum dma_data_direction dir)
{
struct buffer *buffer = attach->dmabuf->priv;
struct buffer_data *buffer_data;
if (!buffer)
return;
pr_debug("%s\n", __func__);
buffer_data = buffer->priv;
sg_dma_address(buffer_data->sgt->sgl) = (~(dma_addr_t)0);
sg_dma_len(buffer_data->sgt->sgl) = 0;
}
/* Called when when ref counter reaches zero! */
static void carveout_release_dmabuf(struct dma_buf *buf)
{
struct buffer *buffer = buf->priv;
struct buffer_data *buffer_data;
if (!buffer)
return;
buffer_data = buffer->priv;
pr_debug("%s %p\n", __func__, buffer_data);
if (!buffer_data)
return;
buffer_data->exported = false;
}
/* Called on file descriptor mmap */
static int carveout_mmap_dmabuf(struct dma_buf *buf, struct vm_area_struct *vma)
{
struct buffer *buffer = buf->priv;
struct buffer_data *buffer_data;
struct scatterlist *sgl;
unsigned long addr;
if (!buffer)
return -EINVAL;
buffer_data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
pr_debug("%s:%d vm_start %#lx vm_end %#lx size %#lx\n",
__func__, __LINE__,
vma->vm_start, vma->vm_end, vma->vm_end - vma->vm_start);
vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
sgl = buffer_data->sgt->sgl;
addr = vma->vm_start;
while (sgl) {
dma_addr_t phys = sg_phys(sgl);
unsigned long pfn = phys >> PAGE_SHIFT;
unsigned int len = sgl->length;
int ret;
if (vma->vm_end < (addr + len)) {
unsigned long size = vma->vm_end - addr;
pr_debug("%s:%d buffer %d (0x%p) truncating len=%#x to size=%#lx\n",
__func__, __LINE__,
buffer->id, buffer, len, size);
WARN(round_up(size, PAGE_SIZE) != size,
"VMA size %#lx not page aligned\n", size);
len = size;
if (!len) /* VM space is smaller than allocation */
break;
}
ret = remap_pfn_range(vma, addr, pfn, len, vma->vm_page_prot);
if (ret)
return ret;
addr += len;
sgl = sg_next(sgl);
}
return 0;
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,19,0)
static void *carveout_kmap_dmabuf(struct dma_buf *buf, unsigned long page)
{
pr_err("%s not supported\n", __func__);
return NULL;
}
#endif
static int carveout_heap_map_km(struct heap *heap, struct buffer *buffer);
static int carveout_heap_unmap_km(struct heap *heap, struct buffer *buffer);
static void *carveout_vmap_dmabuf(struct dma_buf *buf)
{
struct buffer *buffer = buf->priv;
struct heap *heap;
if (!buffer)
return NULL;
heap = buffer->heap;
if (carveout_heap_map_km(heap, buffer))
return NULL;
pr_debug("%s:%d buffer %d kptr 0x%p\n", __func__, __LINE__,
buffer->id, buffer->kptr);
return buffer->kptr;
}
static void carveout_vunmap_dmabuf(struct dma_buf *buf, void *kptr)
{
struct buffer *buffer = buf->priv;
struct heap *heap;
if (!buffer)
return;
heap = buffer->heap;
pr_debug("%s:%d buffer %d kptr 0x%p (0x%p)\n", __func__, __LINE__,
buffer->id, buffer->kptr, kptr);
if (buffer->kptr == kptr)
carveout_heap_unmap_km(heap, buffer);
}
static const struct dma_buf_ops carveout_dmabuf_ops = {
.map_dma_buf = carveout_map_dmabuf,
.unmap_dma_buf = carveout_unmap_dmabuf,
.release = carveout_release_dmabuf,
.mmap = carveout_mmap_dmabuf,
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,12,0)
.kmap_atomic = carveout_kmap_dmabuf,
.kmap = carveout_kmap_dmabuf,
#else
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,19,0)
.map_atomic = carveout_kmap_dmabuf,
.map = carveout_kmap_dmabuf,
#endif
#endif
.vmap = carveout_vmap_dmabuf,
.vunmap = carveout_vunmap_dmabuf,
};
static int carveout_heap_export(struct device *device, struct heap *heap,
size_t size, enum img_mem_attr attr,
struct buffer *buffer, uint64_t* buf_hnd)
{
struct buffer_data *buffer_data = buffer->priv;
struct dma_buf *dma_buf;
int ret, fd;
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,1,0)
DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
#endif
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (!buffer_data)
/* Nothing to export ? */
return -ENOMEM;
if (buffer_data->exported) {
pr_err("%s: already exported!\n", __func__);
return -EBUSY;
}
if (!buffer_data->sgt) {
/* Create for the very first time */
buffer_data->sgt = kzalloc(sizeof(struct sg_table), GFP_KERNEL);
if (!buffer_data->sgt) {
pr_err("%s: failed to allocate sg_table\n", __func__);
return -ENOMEM;
}
ret = sg_alloc_table(buffer_data->sgt, 1, GFP_KERNEL);
if (ret) {
pr_err("%s: sg_alloc_table failed\n", __func__);
goto free_sgt_mem;
}
sg_set_page(buffer_data->sgt->sgl,
pfn_to_page(PFN_DOWN(buffer_data->addr+heap->options.carveout.offs)),
PAGE_ALIGN(size), 0);
/* Store dma info */
if (heap->to_dev_addr)
buffer_data->dma_base = heap->to_dev_addr(&heap->options,
buffer_data->addr+heap->options.carveout.offs);
else
buffer_data->dma_base = buffer_data->addr+heap->options.carveout.offs;
buffer_data->dma_size = PAGE_ALIGN(size);
/* No mapping yet */
sg_dma_address(buffer_data->sgt->sgl) = (~(dma_addr_t)0);
sg_dma_len(buffer_data->sgt->sgl) = 0;
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(3,17,0)
dma_buf = dma_buf_export(buffer_data, &carveout_dmabuf_ops,
size, O_RDWR);
#elif LINUX_VERSION_CODE < KERNEL_VERSION(4,1,0)
dma_buf = dma_buf_export(buffer_data, &carveout_dmabuf_ops,
size, O_RDWR, NULL);
#else
exp_info.ops = &carveout_dmabuf_ops;
exp_info.size = size;
exp_info.flags = O_RDWR;
exp_info.priv = buffer;
exp_info.resv = NULL;
dma_buf = dma_buf_export(&exp_info);
#endif
if (IS_ERR(dma_buf)) {
pr_err("%s:dma_buf_export failed\n", __func__);
ret = PTR_ERR(dma_buf);
return ret;
}
get_dma_buf(dma_buf);
fd = dma_buf_fd(dma_buf, 0);
if (fd < 0) {
pr_err("%s: dma_buf_fd failed\n", __func__);
dma_buf_put(dma_buf);
return -EFAULT;
}
buffer_data->dma_buf = dma_buf;
buffer_data->exported = true;
*buf_hnd = (uint64_t)fd;
return 0;
free_sgt_mem:
kfree(buffer_data->sgt);
buffer_data->sgt = NULL;
return ret;
}
static int carveout_heap_alloc(struct device *device, struct heap *heap,
size_t size, enum img_mem_attr attr,
struct buffer *buffer)
{
struct heap_data *heap_data = heap->priv;
struct buffer_data *buffer_data;
phys_addr_t phys_addr;
size_t pages, page;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
buffer_data = kzalloc(sizeof(struct buffer_data), GFP_KERNEL);
if (!buffer_data)
return -ENOMEM;
pages = size / PAGE_SIZE;
buffer_data->addrs = kmalloc_array(pages, sizeof(uint64_t), GFP_KERNEL);
if (!buffer_data->addrs) {
kfree(buffer_data);
return -ENOMEM;
}
buffer_data->mattr = attr;
buffer_data->addr = gen_pool_alloc(heap_data->pool, size);
if (!buffer_data->addr) {
pr_err("%s gen_pool_alloc failed!\n", __func__);
kfree(buffer_data->addrs);
kfree(buffer_data);
return -ENOMEM;
}
/* The below assigns buffer_data->addr-> 1:1 mapping */
phys_addr = gen_pool_virt_to_phys(heap_data->pool,
buffer_data->addr + heap->options.carveout.offs);
page = 0;
while (page < pages) {
if (trace_physical_pages)
pr_info("%s phys %llx\n",
__func__, (unsigned long long)phys_addr);
buffer_data->addrs[page++] = phys_addr;
phys_addr += PAGE_SIZE;
};
buffer->priv = buffer_data;
pr_debug("%s buffer %d phys %#llx size %zu attrs %x\n", __func__,
buffer->id,
(unsigned long long)buffer_data->addrs[0],
size,
attr);
return 0;
}
static void carveout_heap_free(struct heap *heap, struct buffer *buffer)
{
struct heap_data *heap_data = heap->priv;
struct buffer_data *buffer_data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
/* If forgot to unmap */
if (heap->options.carveout.put_kptr && buffer->kptr) {
heap->options.carveout.put_kptr(buffer->kptr);
buffer->kptr = NULL;
}
if (buffer_data->dma_buf) {
dma_buf_put(buffer_data->dma_buf);
buffer_data->dma_buf->priv = NULL;
}
if (buffer_data->sgt) {
sg_free_table(buffer_data->sgt);
kfree(buffer_data->sgt);
buffer_data->sgt = NULL;
}
if (buffer_data->mapped_vma)
buffer_data->mapped_vma->vm_private_data = NULL;
gen_pool_free(heap_data->pool, buffer_data->addr, buffer->actual_size);
kfree(buffer_data->addrs);
kfree(buffer_data);
}
static void _mmap_open(struct vm_area_struct *vma)
{
struct buffer *buffer = vma->vm_private_data;
struct buffer_data *buffer_data = buffer->priv;
buffer_data->mapped_vma = vma;
pr_debug("%s:%d buffer %d (0x%p) vma:%p\n",
__func__, __LINE__, buffer->id, buffer, vma);
}
static void _mmap_close(struct vm_area_struct *vma)
{
struct buffer *buffer = vma->vm_private_data;
struct buffer_data *buffer_data;
if (!buffer)
return;
buffer_data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p) vma:%p\n",
__func__, __LINE__, buffer->id, buffer, vma);
buffer_data->mapped_vma = NULL;
}
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 11, 0)
static vm_fault_t _mmap_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
#else
static int _mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
#endif
struct buffer *buffer = vma->vm_private_data;
struct buffer_data *buffer_data = buffer->priv;
phys_addr_t phys_addr;
pgoff_t offset;
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 10, 0)
unsigned long addr = vmf->address;
#else
unsigned long addr = (unsigned long)vmf->virtual_address;
#endif
if (trace_mmap_fault) {
pr_debug("%s:%d buffer %d (0x%p) vma:%p\n",
__func__, __LINE__, buffer->id, buffer, vma);
pr_debug("%s:%d vm_start %#lx vm_end %#lx total size %ld\n",
__func__, __LINE__,
vma->vm_start, vma->vm_end,
vma->vm_end - vma->vm_start);
}
offset = (addr - vma->vm_start) >> PAGE_SHIFT;
if (offset > (buffer->actual_size / PAGE_SIZE)) {
pr_err("%s:%d offs:%ld\n",
__func__, __LINE__, offset);
return VM_FAULT_SIGBUS;
}
phys_addr = buffer_data->addrs[0] + (offset * PAGE_SIZE);
if (trace_mmap_fault)
pr_info("%s:%d vmf pgoff %#lx vmf addr %lx offs :%ld phys:%#llx\n",
__func__, __LINE__, vmf->pgoff, addr, offset, phys_addr);
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 20, 0)
return vmf_insert_pfn(vma, addr, phys_addr >> PAGE_SHIFT);
#else
{
int err = vm_insert_pfn(vma, addr, phys_addr >> PAGE_SHIFT);
switch (err) {
case 0:
case -EAGAIN:
case -ERESTARTSYS:
case -EINTR:
case -EBUSY:
return VM_FAULT_NOPAGE;
case -ENOMEM:
return VM_FAULT_OOM;
}
return VM_FAULT_SIGBUS;
}
#endif
}
/* vma ops->fault handler is used to track user space mappings
* (inspired by other gpu/drm drivers from the kernel source tree)
* to properly call cache handling ops when the mapping is destroyed
* (when user calls unmap syscall).
* vma flags are used to choose a correct direction.
* The above facts allows us to do automatic cache flushing/invalidation.
*
* Examples:
* mmap() -> .open -> invalidate buffer cache
* .. read content from buffer
* unmap() -> .close -> do nothing
*
* mmap() -> .open -> do nothing
* .. write content to buffer
* unmap() -> .close -> flush buffer cache
*/
static struct vm_operations_struct carveout_mmap_vm_ops = {
.open = _mmap_open,
.close = _mmap_close,
.fault = _mmap_fault,
};
static int carveout_heap_map_um(struct heap *heap, struct buffer *buffer,
struct vm_area_struct *vma)
{
struct buffer_data *buffer_data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
pr_debug("%s:%d vm_start %#lx vm_end %#lx size %ld\n",
__func__, __LINE__,
vma->vm_start, vma->vm_end, vma->vm_end - vma->vm_start);
/* CACHED by default */
if (buffer_data->mattr & IMG_MEM_ATTR_UNCACHED)
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
else if (buffer_data->mattr & IMG_MEM_ATTR_WRITECOMBINE)
vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
vma->vm_ops = &carveout_mmap_vm_ops;
vma->vm_flags |= VM_PFNMAP;
vma->vm_private_data = buffer;
vma->vm_pgoff = 0;
_mmap_open(vma);
return 0;
}
static int carveout_heap_map_km(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *buffer_data = buffer->priv;
if (buffer->kptr) {
pr_warn("%s called for already mapped buffer %d\n",
__func__, buffer->id);
return 0;
}
if (heap->options.carveout.get_kptr)
buffer->kptr = heap->options.carveout.get_kptr(
buffer_data->addrs[0],
buffer->actual_size,
buffer_data->mattr);
else if (heap->options.carveout.kptr)
buffer->kptr = heap->options.carveout.kptr +
(buffer_data->addrs[0] - heap->options.carveout.phys);
else
return -ENOMEM;
if (!buffer->kptr)
return -ENOMEM;
pr_debug("%s:%d buffer %d (0x%p) kptr 0x%p size:%zu\n", __func__, __LINE__,
buffer->id, buffer, buffer->kptr, buffer->actual_size);
return 0;
}
static int carveout_heap_unmap_km(struct heap *heap, struct buffer *buffer)
{
pr_debug("%s:%d buffer %d (0x%p) kptr 0x%p\n", __func__, __LINE__,
buffer->id, buffer, buffer->kptr);
if (!buffer->kptr) {
pr_warn("%s called for unmapped buffer %d\n",
__func__, buffer->id);
return 0;
}
if (heap->options.carveout.put_kptr)
heap->options.carveout.put_kptr(buffer->kptr);
buffer->kptr = NULL;
return 0;
}
static int carveout_heap_get_page_array(struct heap *heap,
struct buffer *buffer,
uint64_t **addrs)
{
struct buffer_data *buffer_data = buffer->priv;
*addrs = buffer_data->addrs;
return 0;
}
static int carveout_set_offset(struct heap *heap, size_t offs)
{
if (heap->options.carveout.offs > heap->options.carveout.size) {
pr_err("%s offset exceeds size!\n", __func__);
return -EINVAL;
}
heap->options.carveout.offs = offs;
return 0;
}
static void carveout_cache_update(struct vm_area_struct *vma)
{
if (!vma)
return;
pr_debug("%s vma start:%lx end:%lx\n",
__func__, vma->vm_start, vma->vm_end);
#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
/* This function is not exported for modules by ARM kernel */
flush_cache_range(vma, vma->vm_start, vma->vm_end);
#else
/* Tentative for the SFF, this function is exported by the kernel... */
/* vivt_flush_cache_range(vma, vma->vm_start, vma->vm_end); */
#endif
}
static void carveout_sync_cpu_to_dev(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *buffer_data = buffer->priv;
if (!(buffer_data->mattr & IMG_MEM_ATTR_UNCACHED))
carveout_cache_update(buffer_data->mapped_vma);
}
static void carveout_sync_dev_to_cpu(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *buffer_data = buffer->priv;
if (!(buffer_data->mattr & IMG_MEM_ATTR_UNCACHED))
carveout_cache_update(buffer_data->mapped_vma);
}
static void carveout_heap_destroy(struct heap *heap)
{
struct heap_data *heap_data = heap->priv;
pr_debug("%s:%d\n", __func__, __LINE__);
gen_pool_destroy(heap_data->pool);
kfree(heap_data);
}
static struct heap_ops carveout_heap_ops = {
.export = carveout_heap_export,
.alloc = carveout_heap_alloc,
.import = NULL,
.free = carveout_heap_free,
.map_um = carveout_heap_map_um,
.unmap_um = NULL,
.map_km = carveout_heap_map_km,
.unmap_km = carveout_heap_unmap_km,
.get_sg_table = NULL,
.get_page_array = carveout_heap_get_page_array,
.sync_cpu_to_dev = carveout_sync_cpu_to_dev,
.sync_dev_to_cpu = carveout_sync_dev_to_cpu,
.set_offset = carveout_set_offset,
.destroy = carveout_heap_destroy,
};
int img_mem_carveout_init(const struct heap_config *config, struct heap *heap)
{
struct heap_data *heap_data;
unsigned long virt_start;
int ret;
int pool_order = POOL_ALLOC_ORDER_BASE +
heap->options.carveout.pool_order;
if (heap->options.carveout.offs > heap->options.carveout.size) {
pr_err("%s offset exceeds size!\n", __func__);
return -EINVAL;
}
pr_debug("%s phys base:%#llx kptr %p (offs:%llx) size:%zu order:%d\n", __func__,
(unsigned long long)heap->options.carveout.phys,
heap->options.carveout.kptr,
(unsigned long long)heap->options.carveout.offs,
heap->options.carveout.size,
pool_order);
if (config->options.carveout.kptr &&
(heap->options.carveout.put_kptr || heap->options.carveout.get_kptr)) {
pr_err("%s can't use static & dynamic kernel mapping at the same time!\n",
__func__);
return -EINVAL;
}
if (!config->options.carveout.kptr &&
!(heap->options.carveout.put_kptr && heap->options.carveout.get_kptr)) {
pr_warn("%s no kernel mapping method available!\n",
__func__);
return -EINVAL;
}
if (heap->options.carveout.phys & ((1<<pool_order)-1)) {
pr_err("%s phys addr (%#llx) is not aligned to allocation order!\n",
__func__, (unsigned long long)heap->options.carveout.phys);
return -EINVAL;
}
if (heap->options.carveout.size == 0) {
pr_err("%s size cannot be zero!\n", __func__);
return -EINVAL;
}
heap_data = kmalloc(sizeof(struct heap_data), GFP_KERNEL);
if (!heap_data)
return -ENOMEM;
heap_data->pool = gen_pool_create(pool_order, -1);
if (!heap_data->pool) {
pr_err("%s gen_pool_create failed\n", __func__);
ret = -ENOMEM;
goto pool_create_failed;
}
/* Operating in no offset mode -> virtual == phys
* However when physical address == 0 (unlikely) we need to distinguish
* if address returned from gen_pool_alloc is an error or valid address,
* so add a const offset.
*/
virt_start = (unsigned long)heap->options.carveout.phys;
if (!virt_start)
virt_start = 1<<pool_order;
ret = gen_pool_add_virt(heap_data->pool, virt_start,
heap->options.carveout.phys,
heap->options.carveout.size,
-1);
if (ret) {
pr_err("%s gen_pool_add_virt failed\n", __func__);
goto pool_add_failed;
}
heap->ops = &carveout_heap_ops;
heap->priv = heap_data;
return 0;
pool_add_failed:
gen_pool_destroy(heap_data->pool);
pool_create_failed:
kfree(heap_data);
return ret;
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

204
drivers/nna/img_mem/img_mem_coherent.c Normal file
View File

@@ -0,0 +1,204 @@
/*!
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/scatterlist.h>
#include <linux/gfp.h>
#include <linux/vmalloc.h>
#include <linux/dma-mapping.h>
#include <img_mem_man.h>
#include "img_mem_man_priv.h"
struct buffer_data {
void *kptr;
dma_addr_t dma_handle; /* addr returned by dma_alloc_coherent */
uint64_t *addrs; /* array of physical addresses, upcast to 64-bit */
struct device *dev;
size_t size;
};
static int trace_physical_pages;
static int coherent_heap_alloc(struct device *device, struct heap *heap,
size_t size, enum img_mem_attr attr,
struct buffer *buffer)
{
struct buffer_data *buffer_data;
phys_addr_t phys_addr;
size_t pages, page;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
buffer_data = kmalloc(sizeof(struct buffer_data), GFP_KERNEL);
if (!buffer_data)
return -ENOMEM;
pages = size / PAGE_SIZE;
buffer_data->addrs = kmalloc_array(pages, sizeof(uint64_t), GFP_KERNEL);
if (!buffer_data->addrs) {
kfree(buffer_data);
return -ENOMEM;
}
buffer_data->dev = device;
buffer_data->size = size;
buffer_data->kptr = dma_alloc_coherent(device,
size,
&buffer_data->dma_handle,
heap->options.coherent.gfp_flags);
if (!buffer_data->kptr) {
pr_err("%s dma_alloc_coherent failed!\n", __func__);
kfree(buffer_data->addrs);
kfree(buffer_data);
return -ENOMEM;
}
buffer->kptr = (void *)buffer_data->kptr;
page = 0;
phys_addr = buffer_data->dma_handle;
while (page < pages) {
if (trace_physical_pages)
pr_info("%s phys %llx\n",
__func__, (unsigned long long)phys_addr);
buffer_data->addrs[page++] = phys_addr;
phys_addr += PAGE_SIZE;
};
buffer->priv = buffer_data;
pr_debug("%s buffer %d kptr %p phys %#llx size %zu\n", __func__,
buffer->id, buffer->kptr,
(unsigned long long)buffer_data->addrs[0], size);
return 0;
}
static void coherent_heap_free(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *buffer_data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
dma_free_coherent(buffer_data->dev, buffer_data->size,
buffer_data->kptr, buffer_data->dma_handle);
kfree(buffer_data->addrs);
kfree(buffer_data);
}
static int coherent_heap_map_um(struct heap *heap, struct buffer *buffer,
struct vm_area_struct *vma)
{
struct buffer_data *buffer_data = buffer->priv;
unsigned long pfn = *buffer_data->addrs >> PAGE_SHIFT;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
pr_debug("%s:%d vm_start %#lx vm_end %#lx size %ld\n",
__func__, __LINE__,
vma->vm_start, vma->vm_end, vma->vm_end - vma->vm_start);
vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
return remap_pfn_range(vma, vma->vm_start, pfn,
vma->vm_end - vma->vm_start, vma->vm_page_prot);
}
static int coherent_heap_map_km(struct heap *heap, struct buffer *buffer)
{
pr_debug("%s:%d buffer %d (0x%p) kptr 0x%p\n", __func__, __LINE__,
buffer->id, buffer, buffer->kptr);
return 0;
}
static int coherent_heap_unmap_km(struct heap *heap, struct buffer *buffer)
{
pr_debug("%s:%d buffer %d (0x%p) kptr 0x%p\n", __func__, __LINE__,
buffer->id, buffer, buffer->kptr);
return 0;
}
static int coherent_heap_get_page_array(struct heap *heap,
struct buffer *buffer,
uint64_t **addrs)
{
struct buffer_data *buffer_data = buffer->priv;
*addrs = buffer_data->addrs;
return 0;
}
static void coherent_heap_destroy(struct heap *heap)
{
pr_debug("%s:%d\n", __func__, __LINE__);
}
static struct heap_ops coherent_heap_ops = {
.alloc = coherent_heap_alloc,
.import = NULL,
.free = coherent_heap_free,
.map_um = coherent_heap_map_um,
.unmap_um = NULL,
.map_km = coherent_heap_map_km,
.unmap_km = coherent_heap_unmap_km,
.get_sg_table = NULL,
.get_page_array = coherent_heap_get_page_array,
.sync_cpu_to_dev = NULL,
.sync_dev_to_cpu = NULL,
.set_offset = NULL,
.destroy = coherent_heap_destroy,
};
int img_mem_coherent_init(const struct heap_config *config, struct heap *heap)
{
pr_debug("%s gfp:%x\n", __func__,
config->options.coherent.gfp_flags);
heap->ops = &coherent_heap_ops;
return 0;
}

502
drivers/nna/img_mem/img_mem_dmabuf.c Normal file
View File

@@ -0,0 +1,502 @@
/*!
*****************************************************************************
*
* @File img_mem_dmabuf.c
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/scatterlist.h>
#include <linux/gfp.h>
#include <linux/device.h>
#include <linux/vmalloc.h>
#include <linux/dma-buf.h>
#include <linux/scatterlist.h>
#include <linux/dma-mapping.h>
#include <linux/version.h>
#include <img_mem_man.h>
#include "img_mem_man_priv.h"
/* this condition is actually true for kernels < 4.4.100 */
#ifndef PHYS_PFN
#define PHYS_PFN(x) ((unsigned long)((x) >> PAGE_SHIFT))
#endif
static int trace_physical_pages;
static int trace_mmap_fault;
struct buffer_data {
struct dma_buf *dma_buf;
struct dma_buf_attachment *attach;
struct sg_table *sgt;
enum img_mem_attr mattr; /* memory attributes */
struct vm_area_struct *mapped_vma;
};
static int dmabuf_heap_import(struct device *device, struct heap *heap,
size_t size, enum img_mem_attr attr, uint64_t buf_hnd,
struct buffer *buffer)
{
struct buffer_data *data;
int ret;
int buf_fd = (int)buf_hnd;
pr_debug("%s:%d buffer %d (0x%p) buf_fd %d\n", __func__, __LINE__,
buffer->id, buffer, buf_fd);
data = kmalloc(sizeof(struct buffer_data), GFP_KERNEL);
if (!data)
return -ENOMEM;
data->dma_buf = dma_buf_get(buf_fd);
if (IS_ERR_OR_NULL(data->dma_buf)) {
pr_err("%s dma_buf_get fd %d\n", __func__, buf_fd);
ret = -EINVAL;
goto dma_buf_get_failed;
}
pr_debug("%s:%d buffer %d dma_buf %p\n", __func__, __LINE__,
buffer->id, data->dma_buf);
data->attach = dma_buf_attach(data->dma_buf, device);
if (IS_ERR(data->attach)) {
pr_err("%s dma_buf_attach fd %d\n", __func__, buf_fd);
ret = -EINVAL;
goto dma_buf_attach_failed;
}
data->sgt = dma_buf_map_attachment(data->attach, DMA_BIDIRECTIONAL);
if (IS_ERR(data->sgt)) {
pr_err("%s dma_buf_map_attachment fd %d\n", __func__, buf_fd);
ret = -EINVAL;
goto dma_buf_map_failed;
}
if (trace_physical_pages) {
struct scatterlist *sgl = data->sgt->sgl;
while (sgl) {
pr_info("%s:%d phys %#llx length %d (dma_addr:%#llx len:%d)\n",
__func__, __LINE__,
(unsigned long long)sg_phys(sgl), sgl->length,
sg_dma_address(sgl), sg_dma_len(sgl));
sgl = sg_next(sgl);
}
}
data->mattr = attr;
data->mapped_vma = NULL;
buffer->priv = data;
return 0;
dma_buf_map_failed:
dma_buf_detach(data->dma_buf, data->attach);
dma_buf_attach_failed:
dma_buf_put(data->dma_buf);
dma_buf_get_failed:
kfree(data);
return ret;
}
static void dmabuf_heap_free(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (buffer->kptr) {
struct dma_buf *dma_buf = data->dma_buf;
dma_buf_end_cpu_access(dma_buf,
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
0 /* start */,
buffer->actual_size,
#endif
DMA_BIDIRECTIONAL);
dma_buf_vunmap(dma_buf, buffer->kptr);
buffer->kptr = NULL;
}
if (data->mapped_vma)
data->mapped_vma->vm_private_data = NULL;
dma_buf_unmap_attachment(data->attach, data->sgt, DMA_BIDIRECTIONAL);
dma_buf_detach(data->dma_buf, data->attach);
dma_buf_put(data->dma_buf);
kfree(data);
}
static void dmabuf_mmap_open(struct vm_area_struct *vma)
{
struct buffer *buffer = vma->vm_private_data;
struct buffer_data *data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p) vma:%p\n",
__func__, __LINE__, buffer->id, buffer, vma);
if (!(data->mattr & IMG_MEM_ATTR_UNCACHED)) {
enum dma_data_direction dma_dir;
if (vma->vm_flags & VM_WRITE)
dma_dir = DMA_TO_DEVICE;
else
dma_dir = DMA_FROM_DEVICE;
/* User will read the buffer so invalidate D-cache */
dma_buf_begin_cpu_access(data->dma_buf,
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
0 /* start */,
buffer->actual_size,
#endif
dma_dir);
}
data->mapped_vma = vma;
}
static void dmabuf_mmap_close(struct vm_area_struct *vma)
{
struct buffer *buffer = vma->vm_private_data;
struct buffer_data *data;
if (!buffer)
return;
data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p) vma:%p\n",
__func__, __LINE__, buffer->id, buffer, vma);
if (!(data->mattr & IMG_MEM_ATTR_UNCACHED)) {
/* User may have written to the buffer so flush D-cache */
dma_buf_end_cpu_access(data->dma_buf,
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
0 /* start */,
buffer->actual_size,
#endif
DMA_TO_DEVICE);
}
data->mapped_vma = NULL;
}
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 11, 0)
static vm_fault_t dmabuf_mmap_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
#else
static int dmabuf_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
#endif
struct buffer *buffer = vma->vm_private_data;
struct buffer_data *data = buffer->priv;
struct sg_table *sgt = data->sgt;
struct scatterlist *sgl;
pgoff_t curr_offset;
dma_addr_t phys = 0;
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 10, 0)
unsigned long addr = vmf->address;
#else
unsigned long addr = (unsigned long)vmf->virtual_address;
#endif
if (trace_mmap_fault) {
pr_debug("%s:%d buffer %d (0x%p) vma:%p\n",
__func__, __LINE__, buffer->id, buffer, vma);
pr_debug("%s:%d vm_start %#lx vm_end %#lx total size %ld\n",
__func__, __LINE__,
vma->vm_start,
vma->vm_end,
vma->vm_end - vma->vm_start);
}
curr_offset = addr - vma->vm_start;
sgl = sgt->sgl;
while (sgl) {
phys = sg_phys(sgl);
if(curr_offset < sgl->length)
break;
curr_offset -= sgl->length;
sgl = sg_next(sgl);
}
phys += curr_offset; /* set to middle of current block */
if (trace_mmap_fault)
pr_info("%s:%d vmf pgoff:%#lx vmf addr:%lx phys:%#llx\n",
__func__, __LINE__, vmf->pgoff, addr,
(unsigned long long)phys);
{
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 5, 0)
unsigned long pfn = PHYS_PFN(phys);
#else
pfn_t pfn = {
.val = PHYS_PFN(phys)
};
#endif
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 20, 0)
return vmf_insert_mixed(vma, addr, pfn);
#else
{
int err = vm_insert_mixed(vma, addr, pfn);
switch (err) {
case 0:
case -EAGAIN:
case -ERESTARTSYS:
case -EINTR:
case -EBUSY:
return VM_FAULT_NOPAGE;
case -ENOMEM:
return VM_FAULT_OOM;
}
return VM_FAULT_SIGBUS;
}
#endif
}
}
/* vma ops->fault handler is used to track user space mappings
* (inspired by other gpu/drm drivers from the kernel source tree)
* to properly call dma_sync_* ops when the mapping is destroyed
* (when user calls unmap syscall).
* vma flags are used to choose a correct dma mapping.
* By default use DMA_BIDIRECTONAL mapping type (kernel space only).
* The above facts allows us to do automatic cache flushing/invalidation.
*
* Examples:
* mmap() -> .open -> invalidate buffer cache
* .. read content from buffer
* unmap() -> .close -> do nothing
*
* mmap() -> .open -> do nothing
* .. write content to buffer
* unmap() -> .close -> flush buffer cache
*/
static struct vm_operations_struct dmabuf_heap_mmap_vm_ops = {
.open = dmabuf_mmap_open,
.close = dmabuf_mmap_close,
.fault = dmabuf_mmap_fault,
};
static int dmabuf_heap_map_um(struct heap *heap, struct buffer *buffer,
struct vm_area_struct *vma)
{
struct buffer_data *data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
pr_debug("%s:%d vm_start %#lx vm_end %#lx size %ld\n",
__func__, __LINE__,
vma->vm_start, vma->vm_end, vma->vm_end - vma->vm_start);
/* CACHED by default */
if (data->mattr & IMG_MEM_ATTR_WRITECOMBINE)
vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
else if (data->mattr & IMG_MEM_ATTR_UNCACHED)
WARN_ONCE(1, "Uncached not allowed");
/*vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);*/
vma->vm_ops = &dmabuf_heap_mmap_vm_ops;
vma->vm_flags &= ~VM_PFNMAP;
vma->vm_flags |= VM_MIXEDMAP;
vma->vm_private_data = buffer;
vma->vm_pgoff = 0;
dmabuf_mmap_open(vma);
return 0;
}
static int dmabuf_heap_map_km(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *data = buffer->priv;
struct dma_buf *dma_buf = data->dma_buf;
int ret;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (buffer->kptr) {
pr_warn("%s called for already mapped buffer %d\n",
__func__, buffer->id);
return 0;
}
ret = dma_buf_begin_cpu_access(dma_buf,
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
0 /* start */,
buffer->actual_size,
#endif
DMA_BIDIRECTIONAL);
if (ret) {
pr_err("%s begin_cpu_access failed for bufid %d\n", __func__, buffer->id);
return ret;
}
buffer->kptr = dma_buf_vmap(dma_buf);
if (!buffer->kptr) {
pr_err("%s dma_buf_vmap failed!\n", __func__);
return -EFAULT;
}
pr_debug("%s:%d buffer %d vmap to 0x%p\n", __func__, __LINE__,
buffer->id, buffer->kptr);
return 0;
}
static int dmabuf_heap_unmap_km(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *data = buffer->priv;
struct dma_buf *dma_buf = data->dma_buf;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (!buffer->kptr) {
pr_warn("%s called for unmapped buffer %d\n",
__func__, buffer->id);
return 0;
}
dma_buf_end_cpu_access(dma_buf,
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
0 /* start */,
buffer->actual_size,
#endif
DMA_BIDIRECTIONAL);
dma_buf_vunmap(dma_buf, buffer->kptr);
pr_debug("%s:%d buffer %d kunmap from 0x%p\n", __func__, __LINE__,
buffer->id, buffer->kptr);
buffer->kptr = NULL;
return 0;
}
static int dmabuf_get_sg_table(struct heap *heap, struct buffer *buffer,
struct sg_table **sg_table, bool *use_sg_dma)
{
struct buffer_data *data = buffer->priv;
*sg_table = data->sgt;
*use_sg_dma = heap->options.dmabuf.use_sg_dma;
return 0;
}
static void dmabuf_sync_cpu_to_dev(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *data = buffer->priv;
struct dma_buf *dma_buf = data->dma_buf;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (!(data->mattr & IMG_MEM_ATTR_UNCACHED)) {
dma_buf_end_cpu_access(dma_buf,
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
0 /* start */,
buffer->actual_size,
#endif
DMA_TO_DEVICE);
}
}
static void dmabuf_sync_dev_to_cpu(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *data = buffer->priv;
struct dma_buf *dma_buf = data->dma_buf;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (!(data->mattr & IMG_MEM_ATTR_UNCACHED)) {
dma_buf_begin_cpu_access(dma_buf,
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 6, 0)
0 /* start */,
buffer->actual_size,
#endif
DMA_FROM_DEVICE);
}
}
static void dmabuf_heap_destroy(struct heap *heap)
{
pr_debug("%s:%d\n", __func__, __LINE__);
}
static struct heap_ops dmabuf_heap_ops = {
.alloc = NULL,
.import = dmabuf_heap_import,
.free = dmabuf_heap_free,
.map_um = dmabuf_heap_map_um,
.unmap_um = NULL,
.map_km = dmabuf_heap_map_km,
.unmap_km = dmabuf_heap_unmap_km,
.get_sg_table = dmabuf_get_sg_table,
.get_page_array = NULL,
.sync_cpu_to_dev = dmabuf_sync_cpu_to_dev,
.sync_dev_to_cpu = dmabuf_sync_dev_to_cpu,
.set_offset = NULL,
.destroy = dmabuf_heap_destroy,
};
int img_mem_dmabuf_init(const struct heap_config *heap_cfg, struct heap *heap)
{
pr_debug("%s:%d\n", __func__, __LINE__);
heap->ops = &dmabuf_heap_ops;
return 0;
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

266
drivers/nna/img_mem/img_mem_ion.c Normal file
View File

@@ -0,0 +1,266 @@
/*!
*****************************************************************************
*
* @File img_mem_ion.c
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/version.h>
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/scatterlist.h>
#include <linux/device.h>
#include <ion.h>
#include <img_mem_man.h>
#include "img_mem_man_priv.h"
static int trace_physical_pages;
struct buffer_data {
struct ion_client *client;
struct ion_handle *handle;
struct sg_table *sgt;
};
static int ion_heap_import(struct device *device, struct heap *heap,
size_t size, enum img_mem_attr attr, uint64_t buf_hnd,
struct buffer *buffer)
{
struct buffer_data *data;
int ret;
int buf_fd = (int)buf_hnd;
pr_debug("%s:%d buffer %d (0x%p) buf_fd %d\n", __func__, __LINE__,
buffer->id, buffer, buf_fd);
data = kmalloc(sizeof(struct buffer_data), GFP_KERNEL);
if (!data)
return -ENOMEM;
data->client = heap->priv;
data->handle = ion_import_dma_buf(data->client, buf_fd);
if (IS_ERR_OR_NULL(data->handle)) {
pr_err("%s ion_import_dma_buf fd %d\n", __func__, buf_fd);
ret = -EINVAL;
goto ion_import_dma_buf_failed;
}
pr_debug("%s:%d buffer %d ion_handle %p\n", __func__, __LINE__,
buffer->id, data->handle);
data->sgt = ion_sg_table(data->client, data->handle);
if (IS_ERR(data->sgt)) {
pr_err("%s ion_sg_table fd %d\n", __func__, buf_fd);
ret = -EINVAL;
goto ion_sg_table_failed;
}
if (trace_physical_pages) {
struct scatterlist *sgl = data->sgt->sgl;
while (sgl) {
pr_info("%s:%d phys %#llx length %d\n",
__func__, __LINE__,
(unsigned long long)sg_phys(sgl), sgl->length);
sgl = sg_next(sgl);
}
}
buffer->priv = data;
return 0;
ion_sg_table_failed:
ion_free(data->client, data->handle);
ion_import_dma_buf_failed:
kfree(data);
return ret;
}
static void ion_heap_free(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (buffer->kptr)
ion_unmap_kernel(data->client, data->handle);
ion_free(data->client, data->handle);
kfree(data);
}
static int ion_heap_map_um(struct heap *heap, struct buffer *buffer,
struct vm_area_struct *vma)
{
struct buffer_data *data = buffer->priv;
struct scatterlist *sgl;
unsigned long addr;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
pr_debug("%s:%d vm_start %#lx vm_end %#lx size %ld\n",
__func__, __LINE__,
vma->vm_start, vma->vm_end, vma->vm_end - vma->vm_start);
vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot);
sgl = data->sgt->sgl;
addr = vma->vm_start;
while (sgl) {
dma_addr_t phys = sg_phys(sgl); /* sg_dma_address ? */
unsigned long pfn = phys >> PAGE_SHIFT;
unsigned int len = sgl->length;
int ret;
ret = remap_pfn_range(vma, addr, pfn, len, vma->vm_page_prot);
if (ret)
return ret;
addr += len;
sgl = sg_next(sgl);
}
return 0;
}
static int ion_heap_map_km(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (buffer->kptr) {
pr_warn("%s called for already mapped buffer %d\n",
__func__, buffer->id);
return 0;
}
buffer->kptr = ion_map_kernel(data->client, data->handle);
if (!buffer->kptr) {
pr_err("%s ion_map_kernel failed!\n", __func__);
return -EFAULT;
}
pr_debug("%s:%d buffer %d map to 0x%p\n", __func__, __LINE__,
buffer->id, buffer->kptr);
return 0;
}
static int ion_heap_unmap_km(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (!buffer->kptr) {
pr_warn("%s called for unmapped buffer %d\n",
__func__, buffer->id);
return 0;
}
ion_unmap_kernel(data->client, data->handle);
pr_debug("%s:%d buffer %d unmap from 0x%p\n", __func__, __LINE__,
buffer->id, buffer->kptr);
buffer->kptr = NULL;
return 0;
}
static int ion_heap_get_sg_table(struct heap *heap, struct buffer *buffer,
struct sg_table **sg_table, bool *use_sg_dma)
{
struct buffer_data *data = buffer->priv;
*sg_table = data->sgt;
*use_sg_dma = false;
return 0;
}
static void ion_heap_destroy(struct heap *heap)
{
pr_debug("%s:%d\n", __func__, __LINE__);
}
static struct heap_ops ion_heap_ops = {
.alloc = NULL,
.import = ion_heap_import,
.free = ion_heap_free,
.map_um = ion_heap_map_um,
.unmap_um = NULL,
.map_km = ion_heap_map_km,
.unmap_km = ion_heap_unmap_km,
.get_sg_table = ion_heap_get_sg_table,
.get_page_array = NULL,
.sync_cpu_to_dev = NULL,
.sync_dev_to_cpu = NULL,
.set_offset = NULL,
.destroy = ion_heap_destroy,
};
int img_mem_ion_init(const struct heap_config *heap_cfg, struct heap *heap)
{
pr_debug("%s:%d\n", __func__, __LINE__);
if (!heap_cfg->options.ion.client) {
pr_err("%s no ion client defined\n", __func__);
return -EINVAL;
}
heap->ops = &ion_heap_ops;
heap->priv = heap_cfg->options.ion.client;
return 0;
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

2666
drivers/nna/img_mem/img_mem_man.c Normal file
View File

File diff suppressed because it is too large Load Diff

213
drivers/nna/img_mem/img_mem_man_priv.h Normal file
View File

@@ -0,0 +1,213 @@
/*!
*****************************************************************************
*
* @File img_mem_man_priv.h
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef IMG_MEM_MAN_PRIV_H
#define IMG_MEM_MAN_PRIV_H
#include <linux/list.h>
#include <linux/idr.h>
#include <linux/scatterlist.h>
#include <linux/device.h>
#include <img_mem_man.h>
/* Memory context : one per process */
struct mem_ctx {
unsigned id;
struct idr buffers;
struct list_head mmu_ctxs;
/* Used to track memory usage for all buffers and
* separately for MMU page tables only */
size_t mem_usage_max;
size_t mem_usage_curr;
size_t mmu_usage_max;
size_t mmu_usage_curr;
};
/* An MMU mapping of a buffer */
struct mmu_ctx_mapping {
struct mmu_ctx *mmu_ctx;
struct buffer *buffer;
struct imgmmu_map *map;
uint64_t virt_addr;
uint64_t cache_offset;
struct list_head mmu_ctx_entry; /* Entry in <mmu_ctx:mappings> */
struct list_head buffer_entry; /* Entry in <buffer:mappings> */
};
/* mmu context : one per session */
struct mmu_ctx {
unsigned id;
struct device *device;
struct mmu_config config;
struct mem_ctx *mem_ctx; /* for memory allocations */
struct heap *heap; /* for memory allocations */
struct imgmmu_cat *mmu_cat;
struct list_head mappings; /* contains <struct mmu_ctx_mapping> */
struct list_head mem_ctx_entry; /* Entry in <mem_ctx:mmu_ctxs> */
int (*callback_fn)(enum img_mmu_callback_type type, int buf_id,
void *data);
void *callback_data;
unsigned long cache_phys_start;
uint32_t cache_size;
};
#ifdef KERNEL_DMA_FENCE_SUPPORT
struct buffer_fence {
struct dma_fence fence;
spinlock_t lock;
};
#endif
/* Pdump cache info */
struct buffer_pcache {
unsigned int last_offset;
struct scatterlist *last_sgl;
int last_idx;
};
/* buffer : valid in the context of a mem_ctx */
struct buffer {
int id; /* Generated in <mem_ctx:buffers> */
size_t request_size;
size_t actual_size;
struct device *device;
struct mem_ctx *mem_ctx;
struct heap *heap;
struct list_head mappings; /* contains <struct mmu_ctx_mapping> */
void *kptr;
void *priv;
unsigned map_flags;
#ifdef KERNEL_DMA_FENCE_SUPPORT
struct buffer_fence *fence;
#endif
struct buffer_pcache pcache;
};
/* vaa_entry : represents single entry in mmu_vaa */
struct vaa_entry {
struct imgmmu_halloc *alloc;
struct list_head mmu_vaa_entry; /* links to mmu_vaa */
};
/* mmu vaa : one per session */
struct mmu_vaa {
struct device *device;
struct imgmmu_heap *heap;
struct list_head entries; /* contains <struct vaa_entry> */
};
struct heap_ops {
int (*alloc)(struct device *device, struct heap *heap,
size_t size, enum img_mem_attr attr,
struct buffer *buffer);
int (*import)(struct device *device, struct heap *heap,
size_t size, enum img_mem_attr attr, uint64_t buf_hnd,
struct buffer *buffer);
int (*export)(struct device *device, struct heap *heap,
size_t size, enum img_mem_attr attr, struct buffer *buffer,
uint64_t *buf_hnd);
void (*free)(struct heap *heap, struct buffer *buffer);
int (*map_um)(struct heap *heap, struct buffer *buffer,
struct vm_area_struct *vma);
int (*unmap_um)(struct heap *heap, struct buffer *buffer);
int (*map_km)(struct heap *heap, struct buffer *buffer);
int (*unmap_km)(struct heap *heap, struct buffer *buffer);
int (*get_sg_table)(struct heap *heap, struct buffer *buffer,
struct sg_table **sg_table, bool *use_sg_dma);
int (*get_page_array)(struct heap *heap, struct buffer *buffer,
uint64_t **addrs);
void (*sync_cpu_to_dev)(struct heap *heap, struct buffer *buffer);
void (*sync_dev_to_cpu)(struct heap *heap, struct buffer *buffer);
int (*set_offset)(struct heap *heap, size_t offs);
void (*destroy)(struct heap *heap);
};
struct heap {
int id; /* Generated in <mem_man:heaps> */
enum img_mem_heap_type type;
struct heap_ops *ops;
union heap_options options;
phys_addr_t (*to_dev_addr)(union heap_options *opts, phys_addr_t addr);
phys_addr_t (*to_host_addr)(union heap_options *opts, phys_addr_t addr);
bool cache_sync;
enum img_mem_attr alt_cache_attr;
void *priv;
};
int img_mem_unified_init(const struct heap_config *config, struct heap *heap);
int img_mem_coherent_init(const struct heap_config *config, struct heap *heap);
#ifdef CONFIG_DMA_SHARED_BUFFER
int img_mem_dmabuf_init(const struct heap_config *config, struct heap *heap);
#endif
#ifdef CONFIG_ION
int img_mem_ion_init(const struct heap_config *config, struct heap *heap);
#endif
#ifdef CONFIG_GENERIC_ALLOCATOR
int img_mem_carveout_init(const struct heap_config *config, struct heap *heap);
#endif
int img_mem_anonymous_init(const struct heap_config *config, struct heap *heap);
int img_mem_ocm_init(const struct heap_config *config, struct heap *heap);
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 17, 0)
typedef int vm_fault_t;
#endif
#endif /* IMG_MEM_MAN_PRIV_H */
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

173
drivers/nna/img_mem/img_mem_ocm.c Normal file
View File

@@ -0,0 +1,173 @@
/*!
*****************************************************************************
*
* @File img_mem_ocm.c
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/version.h>
#include <linux/module.h>
#include <linux/device.h>
#include <linux/slab.h>
#include <img_mem_man.h>
#include "img_mem_man_priv.h"
static int trace_physical_pages;
struct buffer_data {
uint64_t *addrs; /* array of physical addresses, upcast to 64-bit */
enum img_mem_attr mattr; /* memory attributes */
};
static int ocm_heap_alloc(struct device *device, struct heap *heap,
size_t size, enum img_mem_attr attr,
struct buffer *buffer)
{
struct buffer_data *buffer_data;
phys_addr_t phys_addr;
size_t pages, page;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
if (size > heap->options.ocm.size) {
pr_err("%s requested size bigger than ocm size !\n",
__func__);
return -EINVAL;
}
buffer_data = kzalloc(sizeof(struct buffer_data), GFP_KERNEL);
if (!buffer_data)
return -ENOMEM;
pages = size / PAGE_SIZE;
buffer_data->addrs = kmalloc_array(pages, sizeof(uint64_t), GFP_KERNEL);
if (!buffer_data->addrs) {
kfree(buffer_data);
return -ENOMEM;
}
buffer_data->mattr = attr;
phys_addr = heap->options.ocm.phys;
page = 0;
while (page < pages) {
if (trace_physical_pages)
pr_info("%s phys %llx\n",
__func__, (unsigned long long)phys_addr);
buffer_data->addrs[page++] = phys_addr;
phys_addr += PAGE_SIZE;
};
buffer->priv = buffer_data;
pr_debug("%s buffer %d phys %#llx size %zu attrs %x\n", __func__,
buffer->id,
(unsigned long long)buffer_data->addrs[0],
size,
attr);
return 0;
}
static void ocm_heap_free(struct heap *heap, struct buffer *buffer)
{
struct buffer_data *buffer_data = buffer->priv;
pr_debug("%s:%d buffer %d (0x%p)\n", __func__, __LINE__,
buffer->id, buffer);
kfree(buffer_data->addrs);
kfree(buffer_data);
}
static int ocm_heap_get_page_array(struct heap *heap,
struct buffer *buffer,
uint64_t **addrs)
{
struct buffer_data *buffer_data = buffer->priv;
*addrs = buffer_data->addrs;
return 0;
}
static void ocm_heap_destroy(struct heap *heap)
{
pr_debug("%s:%d\n", __func__, __LINE__);
}
static struct heap_ops ocm_heap_ops = {
.alloc = ocm_heap_alloc,
.import = NULL,
.free = ocm_heap_free,
.map_um = NULL,
.unmap_um = NULL,
.map_km = NULL,
.unmap_km = NULL,
.get_sg_table = NULL,
.get_page_array = ocm_heap_get_page_array,
.sync_cpu_to_dev = NULL,
.sync_dev_to_cpu = NULL,
.set_offset = NULL,
.destroy = ocm_heap_destroy,
};
int img_mem_ocm_init(const struct heap_config *heap_cfg, struct heap *heap)
{
pr_debug("%s phys:%#llx size:%zu attrs:%#x\n", __func__,
(unsigned long long)heap->options.ocm.phys,
heap->options.ocm.size, heap->options.ocm.hattr);
heap->ops = &ocm_heap_ops;
heap->priv = NULL;
return 0;
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

1002
drivers/nna/img_mem/img_mem_unified.c Normal file
View File

File diff suppressed because it is too large Load Diff

189
drivers/nna/img_mem/img_pdump.c Normal file
View File

@@ -0,0 +1,189 @@
/*!
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/mutex.h>
#include <linux/list.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include <stdarg.h>
#include <img_mem_man.h>
#include <vha_drv_common.h>
/*
* create a pdump buffer
* Pdump buffers are currently identified by hard coded number:
* from PDUMP_TXT up to PDUMP_MAX.
* Buffer is allocated using vmalloc, because it might be several MBytes.
*
* If size==0, the buffer can be used for SAB, but not for LDB or TXT.
* (in other words, no memory will be allocated,
* but it will still have a 'length')
*/
struct pdump_buf *img_pdump_create(struct pdump_descr* pdump, uint32_t pdump_num, size_t size)
{
struct pdump_buf *pbuf = &pdump->pbufs[pdump_num];
if (pdump_num >= PDUMP_MAX) {
pr_err("%s: invalid pdump number:%d\n", __func__, pdump_num);
return NULL;
}
if (pbuf->ptr != NULL) {
pr_err("%s: pdump %d already created\n", __func__, pdump_num);
return NULL;
}
pbuf->size = size;
pbuf->len = 0;
if (size == 0)
return pbuf;
pbuf->ptr = vmalloc(size);
pr_debug("%s %d buffer %p size:%zu!\n", __func__,
pdump_num, pbuf->ptr, size);
if (pbuf->ptr == NULL) {
pr_err("%s: failed to create pdump %d\n", __func__, pdump_num);
return NULL;
}
return pbuf;
}
EXPORT_SYMBOL(img_pdump_create);
/*
* append binary data to one of the pdump buffers
*/
int img_pdump_write(struct pdump_descr* pdump, uint32_t pdump_num, const void *ptr, size_t size)
{
struct pdump_buf *pbuf = &pdump->pbufs[pdump_num];
int ret = 0;
if (pdump_num >= PDUMP_MAX || ptr == NULL || pbuf->ptr == NULL)
return -EINVAL;
mutex_lock(&pdump->lock);
if (pbuf->len + size > pbuf->size)
size = pbuf->size - pbuf->len;
if (!size) {
pr_err("%s: no space left in the pdump %d buffer!\n",
__func__, pdump_num);
ret = -ENOSPC;
goto unlock;
}
pr_debug("%s %d buffer len:%zu size:%zu!\n", __func__,
pdump_num, size, pbuf->len);
memcpy(pbuf->ptr + pbuf->len, ptr, size);
pbuf->len += size;
pr_debug("%s end!\n", __func__);
unlock:
mutex_unlock(&pdump->lock);
return ret;
}
EXPORT_SYMBOL(img_pdump_write);
/*
* append a string to the TXT pdump buffer.
* returns the number of bytes printed or error.
*/
__printf(2, 3)
int __img_pdump_printf(struct device* dev, const char *fmt, ...)
{
struct pdump_descr* pdump = vha_pdump_dev_get_drvdata(dev);
struct pdump_buf *pbuf;
va_list ap;
BUG_ON(pdump==NULL);
pbuf = &pdump->pbufs[PDUMP_TXT];
if (pbuf->ptr == NULL)
return -EINVAL;
mutex_lock(&pdump->lock);
va_start(ap, fmt);
if (pbuf->len < pbuf->size) {
#if defined(OSID)
/* Prepend OSID to pdump comments */
if (fmt[0] == '-' && fmt[1] == '-')
pbuf->len += sprintf(pbuf->ptr + pbuf->len,
"-- (OS%d) ", OSID);
#endif
pbuf->len += vsnprintf(pbuf->ptr + pbuf->len,
pbuf->size - pbuf->len,
fmt, ap);
}
/*
* vsnprintf returns the number of bytes that WOULD have been printed
*/
pbuf->len = min(pbuf->size, pbuf->len);
va_end(ap);
mutex_unlock(&pdump->lock);
return pbuf->len;
}
EXPORT_SYMBOL(__img_pdump_printf);
void img_pdump_destroy(struct pdump_descr* pdump)
{
int i;
for (i = 0; i < PDUMP_MAX; i++) {
void *ptr = pdump->pbufs[i].ptr;
pdump->pbufs[i].ptr = NULL;
pr_debug("%s %d buffer %p!\n", __func__, i, ptr);
vfree(ptr);
}
}
EXPORT_SYMBOL(img_pdump_destroy);
/*
* PDUMP generation is disabled until a PDUMP TXT buffer has been created
*/
bool img_pdump_enabled(struct pdump_descr* pdump)
{
return pdump && pdump->pbufs[PDUMP_TXT].ptr != NULL;
}
EXPORT_SYMBOL(img_pdump_enabled);

1449
drivers/nna/img_mem/imgmmu/imgmmu.c Normal file
View File

File diff suppressed because it is too large Load Diff

307
drivers/nna/img_mem/imgmmu/kernel_heap.c Normal file
View File

@@ -0,0 +1,307 @@
/*!
*****************************************************************************
*
* @File kernel_heap.c
* @Description MMU Library: device virtual allocation (heap) implementation
* using gen_alloc from the Linux kernel
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/module.h>
#include <linux/version.h>
#include <linux/init.h>
#include <linux/genalloc.h>
#include <linux/slab.h>
#include "mmulib/heap.h"
/* access to MMU info and error printing function */
#include "mmu_defs.h"
/*#define DEBUG_POOL 1 */
/*
* Internal heap object using genalloc
*/
struct gen_heap {
struct gen_pool *pool;
size_t nalloc;
struct imgmmu_heap hinfo;
};
/*
* The Heap allocation - contains an imgmmu_halloc
* that is given to the caller
*/
struct gen_halloc {
/*
* Associated heap
*/
struct gen_heap *heap;
/*
* MMU lib allocation part
*/
struct imgmmu_halloc virt_mem;
};
/*
* be used for debugging
*/
static void pool_crawler(struct gen_pool *pool,
struct gen_pool_chunk *chunk, void *data) __maybe_unused;
static void pool_crawler(struct gen_pool *pool,
struct gen_pool_chunk *chunk, void *data)
{
#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 12, 0)
unsigned long size = (chunk->end_addr - chunk->start_addr);
#else
unsigned long size = (chunk->end_addr - chunk->start_addr + 1);
#endif
pr_debug("pool 0x%p has chunk 0x%lx to 0x%lx (size = %lu B)\n",
data, chunk->start_addr, chunk->end_addr, size);
}
struct imgmmu_heap *imgmmu_hcreate(uintptr_t vaddr_start,
size_t atom, size_t size, bool guard_band, int *res)
{
struct gen_heap *iheap = NULL;
int min_order = 0; /* log2 of the alloc atom */
size_t isize = atom;
int ret;
uintptr_t start = vaddr_start;
WARN_ON(res == NULL);
WARN_ON(size == 0);
if (size%atom != 0 || (vaddr_start != 0 && vaddr_start%atom != 0)) {
mmu_log_err("Wrong input params: %zu %zu %zu %zu %zu %zu\n",
size, atom, size%atom,
vaddr_start, atom, vaddr_start%atom);
*res = -EINVAL;
return NULL;
}
iheap = kzalloc(sizeof(struct gen_heap), GFP_KERNEL);
if (iheap == NULL) {
*res = -ENOMEM;
return NULL;
}
iheap->nalloc = 0;
/* compute log2 of the alloc atom */
while (isize >>= 1)
min_order++;
/* ugly fix for trouble using gen_pool_alloc() when allocating a block
* gen_pool_alloc() returns 0 on error alought 0 can be a valid
* first virtual address
* therefore all addresses are offseted by the allocation atom
* to insure 0 is the actual error code
*/
if (vaddr_start == 0)
start = vaddr_start+atom; /* otherwise it is vaddr_start */
#if LINUX_VERSION_CODE < KERNEL_VERSION(3, 12, 0)
isize = start + size;
#else
isize = start + size - 1;
#endif
WARN_ON(isize < start); /* too big! it did an overflow */
mmu_log_dbg("create genalloc pool of order %u\n", min_order);
/* -1: not using real inode */
iheap->pool = gen_pool_create(min_order, -1);
if (iheap->pool == NULL) {
*res = -ENOMEM;
mmu_log_err("Failure to create the genalloc pool\n");
kfree(iheap);
return NULL;
}
mmu_log_dbg("pool 0x%p order %u region from 0x%x for %zu bytes\n",
iheap->pool, min_order, start, size);
ret = gen_pool_add(iheap->pool, start, size, -1);
if (ret != 0) {
*res = -EFAULT;
mmu_log_err("Failure to configure the new pool: %d\n", ret);
gen_pool_destroy(iheap->pool);
kfree(iheap);
return NULL;
}
#ifdef DEBUG_POOL
gen_pool_for_each_chunk(iheap->pool, &pool_crawler, iheap->pool);
#endif
iheap->hinfo.vaddr_start = vaddr_start;
iheap->hinfo.atom = atom;
iheap->hinfo.size = size;
iheap->hinfo.guard_band = guard_band;
*res = 0;
return &(iheap->hinfo);
}
struct imgmmu_halloc *imgmmu_hallocate(struct imgmmu_heap *heap,
size_t size, int *res)
{
struct gen_heap *iheap = NULL;
struct gen_halloc *ialloc = NULL;
WARN_ON(res == NULL);
WARN_ON(heap == NULL);
iheap = container_of(heap, struct gen_heap, hinfo);
if (size%heap->atom != 0 || size == 0) {
mmu_log_err("invalid alloc size (0x%zx) for atom:%zu\n",
size, heap->atom);
*res = -EINVAL;
return NULL;
}
ialloc = kzalloc(sizeof(struct gen_halloc), GFP_KERNEL);
if (ialloc == NULL) {
mmu_log_err("failed to allocate internal structure\n");
*res = -ENOMEM;
return NULL;
}
mmu_log_dbg("heap 0x%p alloc %u\n", iheap->pool, size);
/* gen_pool_alloc returns 0 on error
* that is a problem when 1st valid address is 0
* check imgmmu_hcreate for explanations
*/
ialloc->virt_mem.vaddr = gen_pool_alloc(iheap->pool,
/* Take one more atom to create a fake gap between
* virtual addresses, when needed */
iheap->hinfo.guard_band ?
size + iheap->hinfo.atom :
size);
if (ialloc->virt_mem.vaddr == 0) {
mmu_log_err("failed to allocate from gen_pool_alloc\n");
*res = -EFAULT;
kfree(ialloc);
return NULL;
}
mmu_log_dbg(KERN_INFO "heap 0x%p alloc 0x%p %u B atom %u B\n",
iheap->pool, ialloc->virt_mem.vaddr, size, iheap->hinfo.atom);
/* if base address is 0 we applied an offset */
if (iheap->hinfo.vaddr_start == 0)
ialloc->virt_mem.vaddr -= iheap->hinfo.atom;
ialloc->virt_mem.size = size;
ialloc->heap = iheap;
iheap->nalloc++;
#ifdef DEBUG_POOL
gen_pool_for_each_chunk(iheap->pool, &pool_crawler, iheap->pool);
#endif
*res = 0;
return &(ialloc->virt_mem);
}
int imgmmu_hfree(struct imgmmu_halloc *alloc)
{
struct gen_halloc *ialloc = NULL;
uintptr_t addr = 0;
size_t size;
WARN_ON(alloc == NULL);
ialloc = container_of(alloc, struct gen_halloc, virt_mem);
WARN_ON(ialloc->heap == NULL);
WARN_ON(ialloc->heap->pool == NULL);
WARN_ON(ialloc->heap->nalloc == 0);
mmu_log_dbg("heap 0x%p free 0x%p %u B\n",
ialloc->heap->pool, alloc->vaddr, alloc->size);
#ifdef DEBUG_POOL
gen_pool_for_each_chunk(ialloc->heap->pool,
&pool_crawler, ialloc->heap->pool);
#endif
addr = alloc->vaddr;
/* Include a fake gap */
size = ialloc->heap->hinfo.guard_band ?
alloc->size + ialloc->heap->hinfo.atom :
alloc->size;
/* see the explanation in imgmmu_hcreate to know why + atom */
if (ialloc->heap->hinfo.vaddr_start == 0)
addr += ialloc->heap->hinfo.atom;
gen_pool_free(ialloc->heap->pool, addr, size);
ialloc->heap->nalloc--;
kfree(ialloc);
return 0;
}
int imgmmu_hdestroy(struct imgmmu_heap *heap)
{
struct gen_heap *iheap = NULL;
WARN_ON(heap == NULL);
iheap = container_of(heap, struct gen_heap, hinfo);
if (iheap->nalloc > 0) {
mmu_log_err("destroying a heap with non-freed allocation\n");
return -EFAULT;
}
if (iheap->pool != NULL) {
mmu_log_dbg("destroying genalloc pool 0x%p\n", iheap->pool);
gen_pool_destroy(iheap->pool);
}
kfree(iheap);
return 0;
}

142
drivers/nna/img_mem/imgmmu/mmu_defs.h Normal file
View File

@@ -0,0 +1,142 @@
/*!
*****************************************************************************
*
* @File mmu_defs.h
* @Description Internal MMU library header used to define MMU information at
* compilation time and have access to the error printing functions
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef MMU_DEFS_H
#define MMU_DEFS_H
#include <stdarg.h>
/**
* @addtogroup IMGMMU_lib
* @{
*/
/*-------------------------------------------------------------------------
* Following elements are in the IMGMMU_int documentation module
*------------------------------------------------------------------------*/
#ifndef IMGMMU_PHYS_SIZE
/** @brief MMU physical address size in bits */
#define IMGMMU_PHYS_SIZE 40
#endif
#ifndef IMGMMU_VIRT_SIZE
/** @brief MMU virtual address size in bits */
#define IMGMMU_VIRT_SIZE 40
#endif
/** @brief Page size in bytes used at PC & PD (always 4k),
* PT may use variable page size */
#define IMGMMU_PAGE_SIZE 4096u
/** should be log2(IMGMMU_PAGE_SIZE)*3-6 */
#define IMGMMU_CAT_SHIFT 30
/** should be log2(IMGMMU_PAGE_SIZE)*2-3 */
#define IMGMMU_DIR_SHIFT 21
/** should be log2(IMGMMU_PAGE_SIZE) */
#define IMGMMU_PAGE_SHIFT 12
#if IMGMMU_VIRT_SIZE == 40
/**
* @brief maximum number of directories that
* can be stored in the catalogue entry
*/
#define IMGMMU_N_DIR (IMGMMU_PAGE_SIZE/4u)
/**
* @brief maximum number of pagetables that
* can be stored in the directory entry
*/
#define IMGMMU_N_TABLE (IMGMMU_PAGE_SIZE/8u)
/**
* @brief maximum number of page mappings in the pagetable
* for variable page size
*/
#define IMGMMU_N_PAGE (IMGMMU_PAGE_SIZE/ \
((imgmmu_get_page_size()/IMGMMU_PAGE_SIZE)*8u))
#else
/* it is unlikely to change anyway */
#error "need an update for the new virtual address size"
#endif
/** @brief Memory flag used to mark a page mapping as invalid */
#define MMU_FLAG_VALID 0x1
#define MMU_FLAG_INVALID 0x0
/** @brief Other memory flags */
#define MMU_FLAG_READ_ONLY 0x2
/** @brief PTE entry cache bits mask */
#define MMU_PTE_AXCACHE_MASK 0x3C00000000000000UL
/** @brief PTE entry cache bits shift */
#define MMU_PTE_AXCACHE_SHIFT 58
/** @brief PTE entry parity bit shift */
#define MMU_PTE_PARITY_SHIFT 62
/*
* internal printing functions
*/
__printf(4, 5)
void _mmu_log(int err, const char *function, uint32_t line, const char *format, ...);
#define mmu_log_err(...) _mmu_log(1, __func__, __LINE__, __VA_ARGS__)
#define mmu_log_dbg(...)
/*
* #define mmu_log_dbg(...) _mmu_log(0, __func__, __LINE__, __VA_ARGS__)
*/
/**
* @}
*/
/*-------------------------------------------------------------------------
* End of the IMGMMU_int documentation module
*------------------------------------------------------------------------*/
#endif /* MMU_DEFS_H */

159
drivers/nna/img_mem/imgmmu/mmulib/heap.h Normal file
View File

@@ -0,0 +1,159 @@
/*!
*****************************************************************************
*
* @File heap.h
* @Description MMU Library: device virtual allocation (heap)
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef IMGMMU_HEAP_H
#define IMGMMU_HEAP_H
/**
* @defgroup IMGMMU_heap MMU Heap Interface
* @brief The API for the device virtual address Heap - must be implemented
* (see tal_heap.c for an example implementation)
* @ingroup IMGMMU_lib
* @{
*/
/*-----------------------------------------------------------------------------
* Following elements are in the IMGMMU_heap documentation module
*---------------------------------------------------------------------------*/
/** @brief An allocation on a heap. */
struct imgmmu_halloc {
/** @brief Start of the allocation */
uint64_t vaddr;
/** @brief Size in bytes */
size_t size;
};
/**
* @brief A virtual address heap - not directly related to HW MMU directory
* entry
*/
struct imgmmu_heap {
/** @brief Start of device virtual address */
uint64_t vaddr_start;
/** @brief Allocation atom in bytes */
size_t atom;
/** @brief Total size of the heap in bytes */
size_t size;
/** Guard band indicator. */
bool guard_band;
};
/**
* @name Device virtual address allocation (heap management)
* @{
*/
/**
* @brief Create a Heap
*
* @param vaddr_start start of the heap - must be a multiple of atom
* @param atom the minimum possible allocation on the heap in bytes
* - usually related to the system page size
* @param size total size of the heap in bytes
* @param guard_band enables/disables creation of a gap
* between virtual addresses.
* NOTE: The gap has size of atom.
* @param res must be non-NULL - used to give detail about error
*
* @return pointer to the new Heap object and res is 0
* @return NULL and the value of res can be:
* @li -ENOMEM if internal allocation failed
*/
struct imgmmu_heap *imgmmu_hcreate(uintptr_t vaddr_start,
size_t atom, size_t size, bool guard_band, int *res);
/**
* @brief Allocate from a heap
*
* @warning Heap do not relate to each other, therefore one must insure that
* they should not overlap if they should not.
*
* @param heap must not be NULL
* @param size allocation size in bytes
* @param res must be non-NULL - used to give details about error
*
* @return pointer to the new halloc object and res is 0
* @return NULL and the value of res can be:
* @li -EINVAL if the give size is not a multiple of
* heap->atom
* @li -ENOMEM if the internal structure allocation failed
* @li -EFAULT if the internal device memory allocator did not
* find a suitable virtual address
*/
struct imgmmu_halloc *imgmmu_hallocate(struct imgmmu_heap *heap, size_t size,
int *res);
/**
* @brief Liberate an allocation
*
* @return 0
*/
int imgmmu_hfree(struct imgmmu_halloc *alloc);
/**
* @brief Destroy a heap object
*
* @return 0
* @return -EFAULT if the given heap still has attached
* allocation
*/
int imgmmu_hdestroy(struct imgmmu_heap *heap);
/**
* @}
*/
/*-----------------------------------------------------------------------------
* End of the public functions
*---------------------------------------------------------------------------*/
/**
* @}
*/
/*-----------------------------------------------------------------------------
* End of the IMGMMU_heap documentation module
*---------------------------------------------------------------------------*/
#endif /* IMGMMU_HEAP_H */

449
drivers/nna/img_mem/imgmmu/mmulib/mmu.h Normal file
View File

@@ -0,0 +1,449 @@
/*!
*****************************************************************************
*
* @File mmu.h
* @Description MMU Library
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef IMGMMU_MMU_H
#define IMGMMU_MMU_H
#include <linux/scatterlist.h>
/**
* @defgroup IMGMMU_lib The MMU page table manager
* @brief The Memory Mapping Unit page table manager library handles the memory
* hierarchy for a multi-directory MMU.
*
* The library is composed of several elements:
* @li the Page Table management, responsible for managing the device memory
* used for the mapping. This requires some functions from the Allocator to
* access the memory and a virtual address from a Heap.
* @li the Heap interface, that is the SW heap API one can re-implement (or use
* the provided TAL one). This is responsible for choosing a virtual address
* for an allocation.
* @li the Allocator, that is not implemented in this library, is responsible
* for providing physical pages list (when mapping) and a few memory operation
* functions (imgmmu_info).
* An example TAL allocator is provided in this code and can be used when
* running in full user-space with pdumps.
*
* Some pre-processor values can be defined to customise the MMU:
* @li IMGMMU_PHYS_SIZE physical address size of the MMU in bits (default: 40)
* - only used for the default memory write function
* @li IMGMMU_PAGE_SIZE page size in bytes (default: 4096) - not used directly
* in the MMU code, but the allocator should take it into account
* If IMGMMU_PAGE_SIZE is defined the following HAVE TO be defined as well:
* @li IMGMMU_PAGE_SHIFT as log2(IMGMMU_PAGE_SIZE) (default: 12) - used in
* virtual address to determine the position of the page offset
* @li IMGMMU_DIR_SHIFT as log2(IMGMMU_PAGE_SIZE)*2-2 (default: 21) - used in
* virtual address to determine the position of the directory offset
* @li IMGMMU_CAT_SHIFT as log2(IMGMMU_PAGE_SIZE)*3-6 (default: 30) - used in
* virtual address to determine the position of the catalogue offset
*
* @{
*/
/*-----------------------------------------------------------------------------
* Following elements are in the IMGMMU_lib documentation module
*---------------------------------------------------------------------------*/
/**
* @name MMU page table management
* @brief The public functions to use to control the MMU.
* @image html MMU_class.png "MMU structure organisation"
* @{
*/
/*-----------------------------------------------------------------------------
* Following elements are in the public functions
*---------------------------------------------------------------------------*/
/** @brief Opaque type representing an MMU Catalogue page */
struct imgmmu_cat;
/** @brief Opaque type representing an MMU Mapping */
struct imgmmu_map;
struct imgmmu_page;
struct imgmmu_halloc;
/** @brief Define indicating the mmu page type */
#define IMGMMU_PTYPE_PC 0x1
#define IMGMMU_PTYPE_PD 0x2
#define IMGMMU_PTYPE_PT 0x3
/** @brief Bypass phys address translation when using page_write */
#define IMGMMU_BYPASS_ADDR_TRANS 0x80000000
/**
* @brief Pointer to a function implemented by the used allocator to create 1
* page table (used for the MMU mapping - catalogue page and mapping page)
*
* @param type of the mmu page (PC, PD, PT)
*
* This is not done internally to allow the usage of different allocators
*
* @return A populated imgmmu_page structure with the result of the page allocation
* @return NULL if the allocation failed
*
* @see imgmmu_page_free to liberate the page
*/
typedef struct imgmmu_page *(*imgmmu_page_alloc) (void *, unsigned char type);
/**
* @brief Pointer to a function to free the allocated page table used for MMU
* mapping
*
* This is not done internally to allow the usage of different allocators
*
* @see imgmmu_page_alloc to allocate the page
*/
typedef void (*imgmmu_page_free) (struct imgmmu_page *);
/**
* @brief Pointer to a function to update Device memory on non Unified Memory
*/
typedef void (*imgmmu_page_update) (struct imgmmu_page *);
/**
* @brief Write to a device address.
*
* This is not done internally to allow debug operations such a pdumping to
* occur
*
* This function should do all the shifting and masking needed for the used MMU
*
* @param page page to update - asserts it is not NULL
* @param offset in entries (32b word)
* @param write physical address to write
* @param flags bottom part of the entry used as flags for the MMU (including
* valid flag) or IMGMMU_BYPASS_ADDR_TRANS
* @param priv private data passed with map call
*/
typedef void (*imgmmu_page_write) (struct imgmmu_page *page,
unsigned int offset, uint64_t write,
unsigned int flags, void *priv);
/**
* @brief Reads a 32 word on a physical page
*
* This is used only when debug operations occures (e.g. access page table
* entry that caused a page fault)
*
* @param page physical page - asserts it is not NULL
* @param offset in entries (32b word)
* @param priv private data passed with map call
* @return physical address at given offset and flags
*/
typedef uint64_t(*imgmmu_page_read) (struct imgmmu_page *page,
unsigned int offset, void *priv,
unsigned int *flags);
/**
* @brief Callbacks definition structure
*/
struct imgmmu_info {
void *ctx;
/** @brief allocate a physical page used in MMU mapping */
imgmmu_page_alloc page_alloc;
/** @brief liberate a physical page used in MMU mapping */
imgmmu_page_free page_free;
/**
* @brief write a physical address onto a page - optional, if NULL internal
* function is used
*
* The internal function assumes that IMGMMU_PHYS_SIZE is the MMU size.
*
* @note if NULL page_read is also set
*
* @warning The function assumes that the physical page memory is
* accessible
*/
imgmmu_page_write page_write;
/**
* @brief read a physical page offset - optional, if NULL access to page
* table and catalogue entries is not supported
*
* @note If page_write and page_read are NULL then the internal
* function is used.
*
* @warning The function assumes that the physical page memory is
* accessible
*/
imgmmu_page_read page_read;
/**
* @brief update a physical page on device if non UMA - optional, can be
* NULL if update are not needed
*/
imgmmu_page_update page_update;
};
/** @brief Page table entry - used when allocating the MMU pages */
struct imgmmu_page {
/**
* @note Use ui64 instead of uintptr_t to support extended physical address
* on 32b OS
*/
uint64_t phys_addr;
uint64_t virt_base;
uintptr_t cpu_addr;
};
/**
* @brief Access the default specified page size of the MMU (in Bytes)
*/
size_t imgmmu_get_page_size(void);
/**
* @brief Returns entry shift for given type
*/
size_t imgmmu_get_entry_shift(unsigned char type);
/**
* @brief Change the MMU page size in runtime.
*/
int imgmmu_set_page_size(size_t pagesize);
/**
* @brief Access the compilation specified physical size of the MMU (in bits)
*/
size_t imgmmu_get_phys_size(void);
/**
* @brief Access the compilation specified virtual address size of the MMU
* (in bits)
*/
size_t imgmmu_get_virt_size(void);
/**
* @brief Access the CPU page size - similar to PAGE_SIZE macro in Linux
*
* Not directly using PAGE_SIZE because we need a run-time configuration of the
* PAGE_SIZE when running against simulators and different projects define
* PAGE_SIZE in different ways...
*
* The default size is using the PAGE_SIZE macro if defined (or 4kB if not
* defined when running against simulators)
*/
size_t imgmmu_get_cpu_page_size(void);
/**
* @brief Change run-time CPU page size
*
* @warning to use against simulators only! default of imgmmu_get_cpu_page_size()
* is PAGE_SIZE otherwise!
*/
int imgmmu_set_cpu_page_size(size_t pagesize);
/**
* @brief Create a catalogue entry based on a given catalogue configuration
*
* @warning Obviously creation of the catalogue allocates memory - do not call
* while interrupts are disabled
*
* @param info is copied and not modified - contains the functions to
* use to manage page table memory
* @param res where to store the error code, should not be NULL (trapped by
* assert)
*
* @return The opaque handle to the imgmmu_cat object and res to
* 0
* @return NULL in case of an error and res has the value:
* @li -EINVAL if catConfig is NULL or does not
* contain function pointers
* @li -ENOMEM if an internal allocation failed
* @li -EFAULT if the given imgmmu_page_alloc returned NULL
*/
struct imgmmu_cat *imgmmu_cat_create(
const struct imgmmu_info *info,
int *res);
/**
* @brief Destroy the imgmmu_cat - assumes that the HW is not going to access
* the memory any-more
*
* Does not invalidate any memory because it assumes that everything is not
* used any-more
*/
int imgmmu_cat_destroy(struct imgmmu_cat *cat);
/**
* @brief Get access to the page table structure used in the catalogue (to be
* able to write it to registers)
*
* @param cat asserts if cat is NULL
*
* @return the page table structure used
*/
struct imgmmu_page *imgmmu_cat_get_page(struct imgmmu_cat *cat);
/**
* @brief Returns the page table entry value associated with the virtual
* address
*
* @return -1 if the Catalogue's page_read is NULL or if the associate page
* table is invalid in the catalogue map
*
*/
uint64_t imgmmu_cat_get_pte(
struct imgmmu_cat *cat,
uint64_t vaddr);
/**
* @brief Overrides the physical address associated with the virtual address
*
* @return -1 if the Catalogue's page_read is NULL or if the associate page
* table is invalid in the catalogue map
*
*/
uint64_t imgmmu_cat_override_phys_addr(struct imgmmu_cat *cat,
uint64_t vaddr, uint64_t new_phys_addr);
/**
* @brief Create a PageTable mapping for a list of physical pages and device
* virtual address
*
* @warning Mapping can cause memory allocation (missing pages) - do not call
* while interrupts are disabled
*
* @param cat catalogue to use for the mapping
* @param phys_page_list sorted array of physical addresses (ascending order).
* The number of elements is virt_mem->size/MMU_PAGE_SIZE
* @note This array can potentially be big, the caller may need to use vmalloc
* if running the linux kernel (e.g. mapping a 1080p NV12 is 760 entries, 6080
* Bytes - 2 CPU pages needed, fine with kmalloc; 4k NV12 is 3038 entries,
* 24304 Bytes - 6 CPU pages needed, kmalloc would try to find 8 contiguous
* pages which may be problematic if memory is fragmented)
* @param virt_mem associated device virtual address. Given structure is
* copied
* @param map_flags flags to apply on the page (typically 0x2 for Write Only,
* 0x4 for Read Only) - the flag should not set bit 1 as 0x1 is the valid flag.
* @param priv private data to be passed in callback interface
* @param res where to store the error code, should not be NULL
*
* @return The opaque handle to the imgmmu_map object and res to
* 0
* @return NULL in case of an error an res has the value:
* @li -EINVAL if the allocation size is not a multiple of
* IMGMMU_PAGE_SIZE,
* if the given list of page table is too long or not long enough for the
* mapping or
* if the give flags set the invalid bit
* @li -EBUSY if the virtual memory is already mapped
* @li -ENOMEM if an internal allocation failed
* @li -EFAULT if a page creation failed
*/
struct imgmmu_map *imgmmu_cat_map_arr(
struct imgmmu_cat *cat,
uint64_t *phys_page_list,
const struct imgmmu_halloc *virt_mem,
unsigned int map_flags,
void *priv,
int *res);
struct imgmmu_map *imgmmu_cat_map_sg(
struct imgmmu_cat *cat,
struct scatterlist *phys_page_sg,
bool use_sg_dma,
const struct imgmmu_halloc *virt_mem,
unsigned int map_flags,
void *priv,
int *res);
/**
* @brief Un-map the mapped pages (invalidate their entries) and destroy the
* mapping object
*
* This does not destroy the created Page Table (even if they are becoming
* un-used) and does not change the Catalogue valid bits.
*
* @return 0
*/
int imgmmu_cat_unmap(struct imgmmu_map *map);
/**
* @brief Remove the internal Page Table structures and physical pages that
* have no mapping attached to them
*
* @note This function does not have to be used, but can be used to clean some
* un-used memory out and clean the Catalogue valid bits.
*
* @return The number of clean catalogue entries
*/
uint32_t imgmmu_cat_clean(struct imgmmu_cat *cat);
/**
* @brief Get cache bits for PTE entry
*
* @return Cache bits
*/
uint64_t imgmmu_get_pte_cache_bits(uint64_t pte_entry);
/**
* @brief Get parity bit shift of PTE entry
*
* @return Parity bit shift
*/
u8 imgmmu_get_pte_parity_shift(void);
/**
* @brief Set parity for PTE entry
*
* @return Entry with applied parity
*/
void imgmmu_set_pte_parity(uint64_t *pte_entry);
#define IMGMMU_GET_MAX_PAGE_SIZE() (max(imgmmu_get_page_size(), imgmmu_get_cpu_page_size()))
/**
* @}
*/
/*-----------------------------------------------------------------------------
* End of the public functions
*---------------------------------------------------------------------------*/
/**
* @}
*/
/*-----------------------------------------------------------------------------
* End of the IMGMMU_lib documentation module
*---------------------------------------------------------------------------*/
#endif /* IMGMMU_MMU_H */

4
drivers/nna/include/hwdefs/aura_system.h Normal file
View File

@@ -0,0 +1,4 @@
#define _REG_START 0x00000000
#define _VHA_TB_START 0x080000
#define _REG_SIZE 0x00200000
#define _VHA_TB_SIZE 0x00100000

24
drivers/nna/include/hwdefs/gyrus_system.h Normal file
View File

@@ -0,0 +1,24 @@
#define _REG_NNPU_START 0x00000000
#define _REG_START 0x00080000
#define _REG_NNA_START 0x00080000
#define _REG_NNSYS_START 0x00100000
#define _VHA_TB_START 0x000c0000
#define _RGXREG_START 0x00000000
#define _EMUREG_START 0x0000400000
#define _REG_JX_START 0x00000000
#define _REG_JD_START 0x00000000
#define _REG_JTAG_START 0x00000000
#define _REG_META_SD_START 0x00000000
#define _REG_META_TB_START 0x00000000
#define _REG_NNPU_SIZE 0x00080000
#define _REG_SIZE 0x00100000
#define _REG_NNA_SIZE 0x00080000
#define _REG_NNSYS_SIZE 0x00080000
#define _VHA_TB_SIZE 0x00080000
#define _RGXREG_SIZE 0x00080000
#define _EMUREG_SIZE 0x0000104000
#define _REG_JX_SIZE 0xFFFFFFFF
#define _REG_JD_SIZE 0x00000040
#define _REG_JTAG_SIZE 0x00000080
#define _REG_META_SD_SIZE 0x00002800
#define _REG_META_TB_SIZE 0x00010000

4
drivers/nna/include/hwdefs/magna_system.h Normal file
View File

@@ -0,0 +1,4 @@
#define _REG_START 0x00000000
#define _REG_NNA_START 0x00000000
#define _REG_SIZE 0x00080000
#define _REG_NNA_SIZE 0x00080000

4
drivers/nna/include/hwdefs/mirage_system.h Normal file
View File

@@ -0,0 +1,4 @@
#define _REG_START 0x00000000
#define _VHA_TB_START 0x040000
#define _REG_SIZE 0x00100000
#define _VHA_TB_SIZE 0x00100000

355
drivers/nna/include/hwdefs/nn_sys_cr_gyrus.h Normal file
View File

@@ -0,0 +1,355 @@
/*************************************************************************/ /*!
@Title Hardware definition file nn_sys_cr_gyrus.h
@Copyright Copyright (c) Imagination Technologies Ltd. All Rights Reserved
*/ /**************************************************************************/
/* **** Autogenerated C -- do not edit **** */
/*
*/
#ifndef _NN_SYS_CR_GYRUS_H_
#define _NN_SYS_CR_GYRUS_H_
#define NN_SYS_CR_GYRUS_REVISION 1
/*
Register NN_SYS_CR_PRODUCT_ID
*/
#define NN_SYS_CR_PRODUCT_ID (0x0018U)
#define NN_SYS_CR_PRODUCT_ID_MASKFULL (IMG_UINT64_C(0x00000000FFFF0000))
#define NN_SYS_CR_PRODUCT_ID_IMG_PRODUCT_ID_SHIFT (16U)
#define NN_SYS_CR_PRODUCT_ID_IMG_PRODUCT_ID_CLRMSK (IMG_UINT64_C(0XFFFFFFFF0000FFFF))
/*
Register NN_SYS_CR_CORE_ID
*/
#define NN_SYS_CR_CORE_ID (0x0020U)
#define NN_SYS_CR_CORE_ID_MASKFULL (IMG_UINT64_C(0xFFFFFFFFFFFFFFFF))
#define NN_SYS_CR_CORE_ID_BRANCH_ID_SHIFT (48U)
#define NN_SYS_CR_CORE_ID_BRANCH_ID_CLRMSK (IMG_UINT64_C(0X0000FFFFFFFFFFFF))
#define NN_SYS_CR_CORE_ID_VERSION_ID_SHIFT (32U)
#define NN_SYS_CR_CORE_ID_VERSION_ID_CLRMSK (IMG_UINT64_C(0XFFFF0000FFFFFFFF))
#define NN_SYS_CR_CORE_ID_NUMBER_OF_SCALABLE_UNITS_SHIFT (16U)
#define NN_SYS_CR_CORE_ID_NUMBER_OF_SCALABLE_UNITS_CLRMSK (IMG_UINT64_C(0XFFFFFFFF0000FFFF))
#define NN_SYS_CR_CORE_ID_CONFIG_ID_SHIFT (0U)
#define NN_SYS_CR_CORE_ID_CONFIG_ID_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFF0000))
/*
Register NN_SYS_CR_CORE_IP_INTEGRATOR_ID
*/
#define NN_SYS_CR_CORE_IP_INTEGRATOR_ID (0x0028U)
#define NN_SYS_CR_CORE_IP_INTEGRATOR_ID_MASKFULL (IMG_UINT64_C(0x00000000FFFFFFFF))
#define NN_SYS_CR_CORE_IP_INTEGRATOR_ID_VALUE_SHIFT (0U)
#define NN_SYS_CR_CORE_IP_INTEGRATOR_ID_VALUE_CLRMSK (IMG_UINT64_C(0XFFFFFFFF00000000))
/*
Register NN_SYS_CR_CORE_IP_CHANGELIST
*/
#define NN_SYS_CR_CORE_IP_CHANGELIST (0x0030U)
#define NN_SYS_CR_CORE_IP_CHANGELIST_MASKFULL (IMG_UINT64_C(0x00000000FFFFFFFF))
#define NN_SYS_CR_CORE_IP_CHANGELIST_VALUE_SHIFT (0U)
#define NN_SYS_CR_CORE_IP_CHANGELIST_VALUE_CLRMSK (IMG_UINT64_C(0XFFFFFFFF00000000))
#define NN_SYS_CR_CLK_CTRL_MODE_MASK (0x00000003U)
/*
The domain clock is forced off */
#define NN_SYS_CR_CLK_CTRL_MODE_OFF (0x00000000U)
/*
The domain clock is forced on */
#define NN_SYS_CR_CLK_CTRL_MODE_ON (0x00000001U)
/*
Automatic clock gating is active, the domain clock is only on whilst data is being processed */
#define NN_SYS_CR_CLK_CTRL_MODE_AUTO (0x00000002U)
/*
Register NN_SYS_CR_CLK_CTRL
*/
#define NN_SYS_CR_CLK_CTRL (0x0200U)
#define NN_SYS_CR_CLK_CTRL_MASKFULL (IMG_UINT64_C(0x0000000000000003))
#define NN_SYS_CR_CLK_CTRL_NN_SYS_SHIFT (0U)
#define NN_SYS_CR_CLK_CTRL_NN_SYS_CLRMSK (0XFFFFFFFCU)
#define NN_SYS_CR_CLK_CTRL_NN_SYS_OFF (00000000U)
#define NN_SYS_CR_CLK_CTRL_NN_SYS_ON (0X00000001U)
#define NN_SYS_CR_CLK_CTRL_NN_SYS_AUTO (0X00000002U)
/*
Clock is gated and the module is inactive */
#define NN_SYS_CR_CLK_STATUS_MODE_GATED (0x00000000U)
/*
Clock is running */
#define NN_SYS_CR_CLK_STATUS_MODE_RUNNING (0x00000001U)
/*
Register NN_SYS_CR_CLK_STATUS
*/
#define NN_SYS_CR_CLK_STATUS (0x0208U)
#define NN_SYS_CR_CLK_STATUS_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_CLK_STATUS_NN_SYS_SHIFT (0U)
#define NN_SYS_CR_CLK_STATUS_NN_SYS_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_CLK_STATUS_NN_SYS_GATED (00000000U)
#define NN_SYS_CR_CLK_STATUS_NN_SYS_RUNNING (0X00000001U)
/*
Register NN_SYS_CR_EVENT_CLEAR
*/
#define NN_SYS_CR_EVENT_CLEAR (0x0210U)
#define NN_SYS_CR_EVENT_CLEAR_MASKFULL (IMG_UINT64_C(0x0000000000000003))
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_ABORT_SHIFT (1U)
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_ABORT_CLRMSK (0XFFFFFFFDU)
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_ABORT_EN (0X00000002U)
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_COMPLETE_SHIFT (0U)
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_COMPLETE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_COMPLETE_EN (0X00000001U)
/*
Register NN_SYS_CR_EVENT_ENABLE
*/
#define NN_SYS_CR_EVENT_ENABLE (0x0218U)
#define NN_SYS_CR_EVENT_ENABLE_MASKFULL (IMG_UINT64_C(0x0000000000000003))
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_ABORT_SHIFT (1U)
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_ABORT_CLRMSK (0XFFFFFFFDU)
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_ABORT_EN (0X00000002U)
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_COMPLETE_SHIFT (0U)
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_COMPLETE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_COMPLETE_EN (0X00000001U)
/*
Register NN_SYS_CR_EVENT_STATUS
*/
#define NN_SYS_CR_EVENT_STATUS (0x0220U)
#define NN_SYS_CR_EVENT_STATUS_MASKFULL (IMG_UINT64_C(0x0000000000000003))
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_ABORT_SHIFT (1U)
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_ABORT_CLRMSK (0XFFFFFFFDU)
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_ABORT_EN (0X00000002U)
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_COMPLETE_SHIFT (0U)
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_COMPLETE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_COMPLETE_EN (0X00000001U)
/*
Register NN_SYS_CR_IDLE_HYSTERESIS_COUNT
*/
#define NN_SYS_CR_IDLE_HYSTERESIS_COUNT (0x0228U)
#define NN_SYS_CR_IDLE_HYSTERESIS_COUNT_MASKFULL (IMG_UINT64_C(0x000000000000001F))
#define NN_SYS_CR_IDLE_HYSTERESIS_COUNT_NN_SYS_SHIFT (0U)
#define NN_SYS_CR_IDLE_HYSTERESIS_COUNT_NN_SYS_CLRMSK (0XFFFFFFE0U)
/*
Power event type is power down */
#define NN_SYS_CR_POWER_EVENT_MODE_POWER_DOWN (0x00000000U)
/*
Power event type is power up */
#define NN_SYS_CR_POWER_EVENT_MODE_POWER_UP (0x00000001U)
/*
Register NN_SYS_CR_POWER_EVENT
*/
#define NN_SYS_CR_POWER_EVENT (0x0230U)
#define NN_SYS_CR_POWER_EVENT_MASKFULL (IMG_UINT64_C(0x0000000000000033))
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNA_SHIFT (5U)
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNA_CLRMSK (0XFFFFFFDFU)
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNA_EN (0X00000020U)
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNSYS_SHIFT (4U)
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNSYS_CLRMSK (0XFFFFFFEFU)
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNSYS_EN (0X00000010U)
#define NN_SYS_CR_POWER_EVENT_REQUEST_SHIFT (1U)
#define NN_SYS_CR_POWER_EVENT_REQUEST_CLRMSK (0XFFFFFFFDU)
#define NN_SYS_CR_POWER_EVENT_REQUEST_POWER_DOWN (00000000U)
#define NN_SYS_CR_POWER_EVENT_REQUEST_POWER_UP (0X00000002U)
#define NN_SYS_CR_POWER_EVENT_TYPE_SHIFT (0U)
#define NN_SYS_CR_POWER_EVENT_TYPE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_POWER_EVENT_TYPE_EN (0X00000001U)
/*
Register NN_SYS_CR_RESET_CLK_CTRL
*/
#define NN_SYS_CR_RESET_CLK_CTRL (0x0238U)
#define NN_SYS_CR_RESET_CLK_CTRL_MASKFULL (IMG_UINT64_C(0x0000000000000003))
#define NN_SYS_CR_RESET_CLK_CTRL_NN_SYS_SHIFT (0U)
#define NN_SYS_CR_RESET_CLK_CTRL_NN_SYS_CLRMSK (0XFFFFFFFCU)
/*
Register NN_SYS_CR_RESET_CTRL
*/
#define NN_SYS_CR_RESET_CTRL (0x0240U)
#define NN_SYS_CR_RESET_CTRL_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_RESET_CTRL_NN_SYS_SHIFT (0U)
#define NN_SYS_CR_RESET_CTRL_NN_SYS_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_RESET_CTRL_NN_SYS_EN (0X00000001U)
/*
Register NN_SYS_CR_SOCIF_WAKEUP_ENABLE
*/
#define NN_SYS_CR_SOCIF_WAKEUP_ENABLE (0x0248U)
#define NN_SYS_CR_SOCIF_WAKEUP_ENABLE_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_SOCIF_WAKEUP_ENABLE_ALWAYS_SHIFT (0U)
#define NN_SYS_CR_SOCIF_WAKEUP_ENABLE_ALWAYS_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_SOCIF_WAKEUP_ENABLE_ALWAYS_EN (0X00000001U)
/*
Register NN_SYS_CR_AXI_EXACCESS
*/
#define NN_SYS_CR_AXI_EXACCESS (0x0250U)
#define NN_SYS_CR_AXI_EXACCESS_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_AXI_EXACCESS_SOCIF_ENABLE_SHIFT (0U)
#define NN_SYS_CR_AXI_EXACCESS_SOCIF_ENABLE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_AXI_EXACCESS_SOCIF_ENABLE_EN (0X00000001U)
/*
Register NN_SYS_CR_REGBANK_REQUEST_INVALID
*/
#define NN_SYS_CR_REGBANK_REQUEST_INVALID (0x0258U)
#define NN_SYS_CR_REGBANK_REQUEST_INVALID_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_REGBANK_REQUEST_INVALID_FLAG_SHIFT (0U)
#define NN_SYS_CR_REGBANK_REQUEST_INVALID_FLAG_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_REGBANK_REQUEST_INVALID_FLAG_EN (0X00000001U)
/*
Register NN_SYS_CR_NOC_LOWER_ADDR1
*/
#define NN_SYS_CR_NOC_LOWER_ADDR1 (0x0268U)
#define NN_SYS_CR_NOC_LOWER_ADDR1_MASKFULL (IMG_UINT64_C(0x0000000FFFFFFFFF))
#define NN_SYS_CR_NOC_LOWER_ADDR1_LOWER_ADDR_SHIFT (0U)
#define NN_SYS_CR_NOC_LOWER_ADDR1_LOWER_ADDR_CLRMSK (IMG_UINT64_C(0XFFFFFFF000000000))
/*
Register NN_SYS_CR_NOC_UPPER_ADDR1
*/
#define NN_SYS_CR_NOC_UPPER_ADDR1 (0x0278U)
#define NN_SYS_CR_NOC_UPPER_ADDR1_MASKFULL (IMG_UINT64_C(0x0000000FFFFFFFFF))
#define NN_SYS_CR_NOC_UPPER_ADDR1_UPPER_ADDR_SHIFT (0U)
#define NN_SYS_CR_NOC_UPPER_ADDR1_UPPER_ADDR_CLRMSK (IMG_UINT64_C(0XFFFFFFF000000000))
/*
Register NN_SYS_CR_SYS_BUS_DIRECT_ACCESS
*/
#define NN_SYS_CR_SYS_BUS_DIRECT_ACCESS (0x0280U)
#define NN_SYS_CR_SYS_BUS_DIRECT_ACCESS_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_SYS_BUS_DIRECT_ACCESS_SYS_BUS_DIRECT_ACCESS_SHIFT (0U)
#define NN_SYS_CR_SYS_BUS_DIRECT_ACCESS_SYS_BUS_DIRECT_ACCESS_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFFFFE))
#define NN_SYS_CR_SYS_BUS_DIRECT_ACCESS_SYS_BUS_DIRECT_ACCESS_EN (IMG_UINT64_C(0X0000000000000001))
/*
Register NN_SYS_CR_NNPU_ACE_QOS_CTRL
*/
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL (0x02A0U)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_MASKFULL (IMG_UINT64_C(0x000000000000FFFF))
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_CRITICAL_SHIFT (12U)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_CRITICAL_CLRMSK (0XFFFF0FFFU)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_HIGH_SHIFT (8U)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_HIGH_CLRMSK (0XFFFFF0FFU)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_MEDIUM_SHIFT (4U)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_MEDIUM_CLRMSK (0XFFFFFF0FU)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_LOW_SHIFT (0U)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_LOW_CLRMSK (0XFFFFFFF0U)
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_ENUM_PRIORITIES_MASK (0x00000003U)
/*
Low */
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_ENUM_PRIORITIES_LOW (0x00000000U)
/*
Medium */
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_ENUM_PRIORITIES_MEDIUM (0x00000001U)
/*
High */
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_ENUM_PRIORITIES_HIGH (0x00000002U)
/*
Critical */
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_ENUM_PRIORITIES_CRITICAL (0x00000003U)
/*
Register NN_SYS_CR_NNPU_ACE_QOS_SEL
*/
#define NN_SYS_CR_NNPU_ACE_QOS_SEL (0x02A8U)
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MASKFULL (IMG_UINT64_C(0x00000000000000F1))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_SHIFT (6U)
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFFF3F))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_LOW (IMG_UINT64_C(0000000000000000))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_MEDIUM (IMG_UINT64_C(0x0000000000000040))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_HIGH (IMG_UINT64_C(0x0000000000000080))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_CRITICAL (IMG_UINT64_C(0x00000000000000c0))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_SHIFT (4U)
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFFFCF))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_LOW (IMG_UINT64_C(0000000000000000))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_MEDIUM (IMG_UINT64_C(0x0000000000000010))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_HIGH (IMG_UINT64_C(0x0000000000000020))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_CRITICAL (IMG_UINT64_C(0x0000000000000030))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NNPU_QOS_ENABLE_SHIFT (0U)
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NNPU_QOS_ENABLE_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFFFFE))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NNPU_QOS_ENABLE_EN (IMG_UINT64_C(0X0000000000000001))
/*
Register NN_SYS_CR_RTM_CTRL
*/
#define NN_SYS_CR_RTM_CTRL (0x1000U)
#define NN_SYS_CR_RTM_CTRL_MASKFULL (IMG_UINT64_C(0x00000000C0000FF8))
#define NN_SYS_CR_RTM_CTRL_RTM_ENABLE_SHIFT (31U)
#define NN_SYS_CR_RTM_CTRL_RTM_ENABLE_CLRMSK (0X7FFFFFFFU)
#define NN_SYS_CR_RTM_CTRL_RTM_ENABLE_EN (0X80000000U)
#define NN_SYS_CR_RTM_CTRL_RTM_CHECK_SHIFT (30U)
#define NN_SYS_CR_RTM_CTRL_RTM_CHECK_CLRMSK (0XBFFFFFFFU)
#define NN_SYS_CR_RTM_CTRL_RTM_CHECK_EN (0X40000000U)
#define NN_SYS_CR_RTM_CTRL_RTM_SELECTOR_SHIFT (3U)
#define NN_SYS_CR_RTM_CTRL_RTM_SELECTOR_CLRMSK (0XFFFFF007U)
/*
Register NN_SYS_CR_RTM_DATA
*/
#define NN_SYS_CR_RTM_DATA (0x1008U)
#define NN_SYS_CR_RTM_DATA_MASKFULL (IMG_UINT64_C(0x00000000FFFFFFFF))
#define NN_SYS_CR_RTM_DATA_RTM_DATA_SHIFT (0U)
#define NN_SYS_CR_RTM_DATA_RTM_DATA_CLRMSK (00000000U)
/*
Register NN_SYS_CR_SOCIF_BUS_UNTRUSTED
*/
#define NN_SYS_CR_SOCIF_BUS_UNTRUSTED (0x1A000U)
#define NN_SYS_CR_SOCIF_BUS_UNTRUSTED_MASKFULL (IMG_UINT64_C(0xFFFFFFFFFFFFFFFF))
#define NN_SYS_CR_SOCIF_BUS_UNTRUSTED_VALUE_SHIFT (0U)
#define NN_SYS_CR_SOCIF_BUS_UNTRUSTED_VALUE_CLRMSK (IMG_UINT64_C(0000000000000000))
/*
Register NN_SYS_CR_SOCIF_BUS_SECURE
*/
#define NN_SYS_CR_SOCIF_BUS_SECURE (0x1A100U)
#define NN_SYS_CR_SOCIF_BUS_SECURE_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_SOCIF_BUS_SECURE_ENABLE_SHIFT (0U)
#define NN_SYS_CR_SOCIF_BUS_SECURE_ENABLE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_SOCIF_BUS_SECURE_ENABLE_EN (0X00000001U)
#endif /* _NN_SYS_CR_GYRUS_H_ */
/*****************************************************************************
End of file (nn_sys_cr_gyrus.h)
*****************************************************************************/

364
drivers/nna/include/hwdefs/nn_sys_cr_vagus.h Normal file
View File

@@ -0,0 +1,364 @@
/*************************************************************************/ /*!
@Title Hardware definition file nn_sys_cr_vagus.h
@Copyright Copyright (c) Imagination Technologies Ltd. All Rights Reserved
*/ /**************************************************************************/
/* **** Autogenerated C -- do not edit **** */
/*
*/
#ifndef _NN_SYS_CR_VAGUS_H_
#define _NN_SYS_CR_VAGUS_H_
#define NN_SYS_CR_VAGUS_REVISION 1
/*
Register NN_SYS_CR_PRODUCT_ID
*/
#define NN_SYS_CR_PRODUCT_ID (0x0018U)
#define NN_SYS_CR_PRODUCT_ID_MASKFULL (IMG_UINT64_C(0x00000000FFFF0000))
#define NN_SYS_CR_PRODUCT_ID_IMG_PRODUCT_ID_SHIFT (16U)
#define NN_SYS_CR_PRODUCT_ID_IMG_PRODUCT_ID_CLRMSK (IMG_UINT64_C(0XFFFFFFFF0000FFFF))
/*
Register NN_SYS_CR_CORE_ID
*/
#define NN_SYS_CR_CORE_ID (0x0020U)
#define NN_SYS_CR_CORE_ID_MASKFULL (IMG_UINT64_C(0xFFFFFFFFFFFFFFFF))
#define NN_SYS_CR_CORE_ID_BRANCH_ID_SHIFT (48U)
#define NN_SYS_CR_CORE_ID_BRANCH_ID_CLRMSK (IMG_UINT64_C(0X0000FFFFFFFFFFFF))
#define NN_SYS_CR_CORE_ID_VERSION_ID_SHIFT (32U)
#define NN_SYS_CR_CORE_ID_VERSION_ID_CLRMSK (IMG_UINT64_C(0XFFFF0000FFFFFFFF))
#define NN_SYS_CR_CORE_ID_NUMBER_OF_SCALABLE_UNITS_SHIFT (16U)
#define NN_SYS_CR_CORE_ID_NUMBER_OF_SCALABLE_UNITS_CLRMSK (IMG_UINT64_C(0XFFFFFFFF0000FFFF))
#define NN_SYS_CR_CORE_ID_CONFIG_ID_SHIFT (0U)
#define NN_SYS_CR_CORE_ID_CONFIG_ID_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFF0000))
/*
Register NN_SYS_CR_CORE_IP_INTEGRATOR_ID
*/
#define NN_SYS_CR_CORE_IP_INTEGRATOR_ID (0x0028U)
#define NN_SYS_CR_CORE_IP_INTEGRATOR_ID_MASKFULL (IMG_UINT64_C(0x00000000FFFFFFFF))
#define NN_SYS_CR_CORE_IP_INTEGRATOR_ID_VALUE_SHIFT (0U)
#define NN_SYS_CR_CORE_IP_INTEGRATOR_ID_VALUE_CLRMSK (IMG_UINT64_C(0XFFFFFFFF00000000))
/*
Register NN_SYS_CR_CORE_IP_CHANGELIST
*/
#define NN_SYS_CR_CORE_IP_CHANGELIST (0x0030U)
#define NN_SYS_CR_CORE_IP_CHANGELIST_MASKFULL (IMG_UINT64_C(0x00000000FFFFFFFF))
#define NN_SYS_CR_CORE_IP_CHANGELIST_VALUE_SHIFT (0U)
#define NN_SYS_CR_CORE_IP_CHANGELIST_VALUE_CLRMSK (IMG_UINT64_C(0XFFFFFFFF00000000))
/*
Register NN_SYS_CR_CORE_IP_CONFIG
*/
#define NN_SYS_CR_CORE_IP_CONFIG (0x0038U)
#define NN_SYS_CR_CORE_IP_CONFIG_MASKFULL (IMG_UINT64_C(0x0000000000000FFF))
#define NN_SYS_CR_CORE_IP_CONFIG_NN_SYS_OCM_RAM_SIZE_4KB_SHIFT (0U)
#define NN_SYS_CR_CORE_IP_CONFIG_NN_SYS_OCM_RAM_SIZE_4KB_CLRMSK (0XFFFFF000U)
#define NN_SYS_CR_CLK_CTRL_MODE_MASK (0x00000003U)
/*
The domain clock is forced off */
#define NN_SYS_CR_CLK_CTRL_MODE_OFF (0x00000000U)
/*
The domain clock is forced on */
#define NN_SYS_CR_CLK_CTRL_MODE_ON (0x00000001U)
/*
Automatic clock gating is active, the domain clock is only on whilst data is being processed */
#define NN_SYS_CR_CLK_CTRL_MODE_AUTO (0x00000002U)
/*
Register NN_SYS_CR_CLK_CTRL
*/
#define NN_SYS_CR_CLK_CTRL (0x0200U)
#define NN_SYS_CR_CLK_CTRL_MASKFULL (IMG_UINT64_C(0x0000000000000003))
#define NN_SYS_CR_CLK_CTRL_NN_SYS_SHIFT (0U)
#define NN_SYS_CR_CLK_CTRL_NN_SYS_CLRMSK (0XFFFFFFFCU)
#define NN_SYS_CR_CLK_CTRL_NN_SYS_OFF (00000000U)
#define NN_SYS_CR_CLK_CTRL_NN_SYS_ON (0X00000001U)
#define NN_SYS_CR_CLK_CTRL_NN_SYS_AUTO (0X00000002U)
/*
Clock is gated and the module is inactive */
#define NN_SYS_CR_CLK_STATUS_MODE_GATED (0x00000000U)
/*
Clock is running */
#define NN_SYS_CR_CLK_STATUS_MODE_RUNNING (0x00000001U)
/*
Register NN_SYS_CR_CLK_STATUS
*/
#define NN_SYS_CR_CLK_STATUS (0x0208U)
#define NN_SYS_CR_CLK_STATUS_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_CLK_STATUS_NN_SYS_SHIFT (0U)
#define NN_SYS_CR_CLK_STATUS_NN_SYS_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_CLK_STATUS_NN_SYS_GATED (00000000U)
#define NN_SYS_CR_CLK_STATUS_NN_SYS_RUNNING (0X00000001U)
/*
Register NN_SYS_CR_EVENT_CLEAR
*/
#define NN_SYS_CR_EVENT_CLEAR (0x0210U)
#define NN_SYS_CR_EVENT_CLEAR_MASKFULL (IMG_UINT64_C(0x0000000000000003))
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_ABORT_SHIFT (1U)
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_ABORT_CLRMSK (0XFFFFFFFDU)
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_ABORT_EN (0X00000002U)
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_COMPLETE_SHIFT (0U)
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_COMPLETE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_EVENT_CLEAR_NN_SYS_PWR_COMPLETE_EN (0X00000001U)
/*
Register NN_SYS_CR_EVENT_ENABLE
*/
#define NN_SYS_CR_EVENT_ENABLE (0x0218U)
#define NN_SYS_CR_EVENT_ENABLE_MASKFULL (IMG_UINT64_C(0x0000000000000003))
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_ABORT_SHIFT (1U)
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_ABORT_CLRMSK (0XFFFFFFFDU)
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_ABORT_EN (0X00000002U)
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_COMPLETE_SHIFT (0U)
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_COMPLETE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_EVENT_ENABLE_NN_SYS_PWR_COMPLETE_EN (0X00000001U)
/*
Register NN_SYS_CR_EVENT_STATUS
*/
#define NN_SYS_CR_EVENT_STATUS (0x0220U)
#define NN_SYS_CR_EVENT_STATUS_MASKFULL (IMG_UINT64_C(0x0000000000000003))
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_ABORT_SHIFT (1U)
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_ABORT_CLRMSK (0XFFFFFFFDU)
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_ABORT_EN (0X00000002U)
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_COMPLETE_SHIFT (0U)
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_COMPLETE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_EVENT_STATUS_NN_SYS_PWR_COMPLETE_EN (0X00000001U)
/*
Register NN_SYS_CR_IDLE_HYSTERESIS_COUNT
*/
#define NN_SYS_CR_IDLE_HYSTERESIS_COUNT (0x0228U)
#define NN_SYS_CR_IDLE_HYSTERESIS_COUNT_MASKFULL (IMG_UINT64_C(0x000000000000001F))
#define NN_SYS_CR_IDLE_HYSTERESIS_COUNT_NN_SYS_SHIFT (0U)
#define NN_SYS_CR_IDLE_HYSTERESIS_COUNT_NN_SYS_CLRMSK (0XFFFFFFE0U)
/*
Power event type is power down */
#define NN_SYS_CR_POWER_EVENT_MODE_POWER_DOWN (0x00000000U)
/*
Power event type is power up */
#define NN_SYS_CR_POWER_EVENT_MODE_POWER_UP (0x00000001U)
/*
Register NN_SYS_CR_POWER_EVENT
*/
#define NN_SYS_CR_POWER_EVENT (0x0230U)
#define NN_SYS_CR_POWER_EVENT_MASKFULL (IMG_UINT64_C(0x0000000000000033))
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNA_SHIFT (5U)
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNA_CLRMSK (0XFFFFFFDFU)
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNA_EN (0X00000020U)
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNSYS_SHIFT (4U)
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNSYS_CLRMSK (0XFFFFFFEFU)
#define NN_SYS_CR_POWER_EVENT_DOMAIN_NNSYS_EN (0X00000010U)
#define NN_SYS_CR_POWER_EVENT_REQUEST_SHIFT (1U)
#define NN_SYS_CR_POWER_EVENT_REQUEST_CLRMSK (0XFFFFFFFDU)
#define NN_SYS_CR_POWER_EVENT_REQUEST_POWER_DOWN (00000000U)
#define NN_SYS_CR_POWER_EVENT_REQUEST_POWER_UP (0X00000002U)
#define NN_SYS_CR_POWER_EVENT_TYPE_SHIFT (0U)
#define NN_SYS_CR_POWER_EVENT_TYPE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_POWER_EVENT_TYPE_EN (0X00000001U)
/*
Register NN_SYS_CR_RESET_CLK_CTRL
*/
#define NN_SYS_CR_RESET_CLK_CTRL (0x0238U)
#define NN_SYS_CR_RESET_CLK_CTRL_MASKFULL (IMG_UINT64_C(0x0000000000000003))
#define NN_SYS_CR_RESET_CLK_CTRL_NN_SYS_SHIFT (0U)
#define NN_SYS_CR_RESET_CLK_CTRL_NN_SYS_CLRMSK (0XFFFFFFFCU)
/*
Register NN_SYS_CR_RESET_CTRL
*/
#define NN_SYS_CR_RESET_CTRL (0x0240U)
#define NN_SYS_CR_RESET_CTRL_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_RESET_CTRL_NN_SYS_SHIFT (0U)
#define NN_SYS_CR_RESET_CTRL_NN_SYS_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_RESET_CTRL_NN_SYS_EN (0X00000001U)
/*
Register NN_SYS_CR_SOCIF_WAKEUP_ENABLE
*/
#define NN_SYS_CR_SOCIF_WAKEUP_ENABLE (0x0248U)
#define NN_SYS_CR_SOCIF_WAKEUP_ENABLE_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_SOCIF_WAKEUP_ENABLE_ALWAYS_SHIFT (0U)
#define NN_SYS_CR_SOCIF_WAKEUP_ENABLE_ALWAYS_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_SOCIF_WAKEUP_ENABLE_ALWAYS_EN (0X00000001U)
/*
Register NN_SYS_CR_AXI_EXACCESS
*/
#define NN_SYS_CR_AXI_EXACCESS (0x0250U)
#define NN_SYS_CR_AXI_EXACCESS_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_AXI_EXACCESS_SOCIF_ENABLE_SHIFT (0U)
#define NN_SYS_CR_AXI_EXACCESS_SOCIF_ENABLE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_AXI_EXACCESS_SOCIF_ENABLE_EN (0X00000001U)
/*
Register NN_SYS_CR_REGBANK_REQUEST_INVALID
*/
#define NN_SYS_CR_REGBANK_REQUEST_INVALID (0x0258U)
#define NN_SYS_CR_REGBANK_REQUEST_INVALID_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_REGBANK_REQUEST_INVALID_FLAG_SHIFT (0U)
#define NN_SYS_CR_REGBANK_REQUEST_INVALID_FLAG_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_REGBANK_REQUEST_INVALID_FLAG_EN (0X00000001U)
/*
Register NN_SYS_CR_NOC_LOWER_ADDR1
*/
#define NN_SYS_CR_NOC_LOWER_ADDR1 (0x0268U)
#define NN_SYS_CR_NOC_LOWER_ADDR1_MASKFULL (IMG_UINT64_C(0x0000000FFFFFFFFF))
#define NN_SYS_CR_NOC_LOWER_ADDR1_LOWER_ADDR_SHIFT (0U)
#define NN_SYS_CR_NOC_LOWER_ADDR1_LOWER_ADDR_CLRMSK (IMG_UINT64_C(0XFFFFFFF000000000))
/*
Register NN_SYS_CR_NOC_UPPER_ADDR1
*/
#define NN_SYS_CR_NOC_UPPER_ADDR1 (0x0278U)
#define NN_SYS_CR_NOC_UPPER_ADDR1_MASKFULL (IMG_UINT64_C(0x0000000FFFFFFFFF))
#define NN_SYS_CR_NOC_UPPER_ADDR1_UPPER_ADDR_SHIFT (0U)
#define NN_SYS_CR_NOC_UPPER_ADDR1_UPPER_ADDR_CLRMSK (IMG_UINT64_C(0XFFFFFFF000000000))
/*
Register NN_SYS_CR_SYS_BUS_DIRECT_ACCESS
*/
#define NN_SYS_CR_SYS_BUS_DIRECT_ACCESS (0x0280U)
#define NN_SYS_CR_SYS_BUS_DIRECT_ACCESS_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_SYS_BUS_DIRECT_ACCESS_SYS_BUS_DIRECT_ACCESS_SHIFT (0U)
#define NN_SYS_CR_SYS_BUS_DIRECT_ACCESS_SYS_BUS_DIRECT_ACCESS_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFFFFE))
#define NN_SYS_CR_SYS_BUS_DIRECT_ACCESS_SYS_BUS_DIRECT_ACCESS_EN (IMG_UINT64_C(0X0000000000000001))
/*
Register NN_SYS_CR_NNPU_ACE_QOS_CTRL
*/
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL (0x02A0U)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_MASKFULL (IMG_UINT64_C(0x000000000000FFFF))
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_CRITICAL_SHIFT (12U)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_CRITICAL_CLRMSK (0XFFFF0FFFU)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_HIGH_SHIFT (8U)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_HIGH_CLRMSK (0XFFFFF0FFU)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_MEDIUM_SHIFT (4U)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_MEDIUM_CLRMSK (0XFFFFFF0FU)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_LOW_SHIFT (0U)
#define NN_SYS_CR_NNPU_ACE_QOS_CTRL_LOW_CLRMSK (0XFFFFFFF0U)
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_ENUM_PRIORITIES_MASK (0x00000003U)
/*
Low */
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_ENUM_PRIORITIES_LOW (0x00000000U)
/*
Medium */
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_ENUM_PRIORITIES_MEDIUM (0x00000001U)
/*
High */
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_ENUM_PRIORITIES_HIGH (0x00000002U)
/*
Critical */
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_ENUM_PRIORITIES_CRITICAL (0x00000003U)
/*
Register NN_SYS_CR_NNPU_ACE_QOS_SEL
*/
#define NN_SYS_CR_NNPU_ACE_QOS_SEL (0x02A8U)
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MASKFULL (IMG_UINT64_C(0x00000000000000F1))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_SHIFT (6U)
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFFF3F))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_LOW (IMG_UINT64_C(0000000000000000))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_MEDIUM (IMG_UINT64_C(0x0000000000000040))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_HIGH (IMG_UINT64_C(0x0000000000000080))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_MMU_CRITICAL (IMG_UINT64_C(0x00000000000000c0))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_SHIFT (4U)
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFFFCF))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_LOW (IMG_UINT64_C(0000000000000000))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_MEDIUM (IMG_UINT64_C(0x0000000000000010))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_HIGH (IMG_UINT64_C(0x0000000000000020))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NON_MMU_CRITICAL (IMG_UINT64_C(0x0000000000000030))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NNPU_QOS_ENABLE_SHIFT (0U)
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NNPU_QOS_ENABLE_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFFFFE))
#define NN_SYS_CR_NNPU_ACE_QOS_SEL_NNPU_QOS_ENABLE_EN (IMG_UINT64_C(0X0000000000000001))
/*
Register NN_SYS_CR_RTM_CTRL
*/
#define NN_SYS_CR_RTM_CTRL (0x1000U)
#define NN_SYS_CR_RTM_CTRL_MASKFULL (IMG_UINT64_C(0x00000000C0000FF8))
#define NN_SYS_CR_RTM_CTRL_RTM_ENABLE_SHIFT (31U)
#define NN_SYS_CR_RTM_CTRL_RTM_ENABLE_CLRMSK (0X7FFFFFFFU)
#define NN_SYS_CR_RTM_CTRL_RTM_ENABLE_EN (0X80000000U)
#define NN_SYS_CR_RTM_CTRL_RTM_CHECK_SHIFT (30U)
#define NN_SYS_CR_RTM_CTRL_RTM_CHECK_CLRMSK (0XBFFFFFFFU)
#define NN_SYS_CR_RTM_CTRL_RTM_CHECK_EN (0X40000000U)
#define NN_SYS_CR_RTM_CTRL_RTM_SELECTOR_SHIFT (3U)
#define NN_SYS_CR_RTM_CTRL_RTM_SELECTOR_CLRMSK (0XFFFFF007U)
/*
Register NN_SYS_CR_RTM_DATA
*/
#define NN_SYS_CR_RTM_DATA (0x1008U)
#define NN_SYS_CR_RTM_DATA_MASKFULL (IMG_UINT64_C(0x00000000FFFFFFFF))
#define NN_SYS_CR_RTM_DATA_RTM_DATA_SHIFT (0U)
#define NN_SYS_CR_RTM_DATA_RTM_DATA_CLRMSK (00000000U)
/*
Register NN_SYS_CR_SOCIF_BUS_UNTRUSTED
*/
#define NN_SYS_CR_SOCIF_BUS_UNTRUSTED (0x1A000U)
#define NN_SYS_CR_SOCIF_BUS_UNTRUSTED_MASKFULL (IMG_UINT64_C(0xFFFFFFFFFFFFFFFF))
#define NN_SYS_CR_SOCIF_BUS_UNTRUSTED_VALUE_SHIFT (0U)
#define NN_SYS_CR_SOCIF_BUS_UNTRUSTED_VALUE_CLRMSK (IMG_UINT64_C(0000000000000000))
/*
Register NN_SYS_CR_SOCIF_BUS_SECURE
*/
#define NN_SYS_CR_SOCIF_BUS_SECURE (0x1A100U)
#define NN_SYS_CR_SOCIF_BUS_SECURE_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define NN_SYS_CR_SOCIF_BUS_SECURE_ENABLE_SHIFT (0U)
#define NN_SYS_CR_SOCIF_BUS_SECURE_ENABLE_CLRMSK (0XFFFFFFFEU)
#define NN_SYS_CR_SOCIF_BUS_SECURE_ENABLE_EN (0X00000001U)
#endif /* _NN_SYS_CR_VAGUS_H_ */
/*****************************************************************************
End of file (nn_sys_cr_vagus.h)
*****************************************************************************/

8
drivers/nna/include/hwdefs/vagus_system.h Normal file
View File

@@ -0,0 +1,8 @@
#define _REG_START 0x00000000
#define _REG_NNA_START 0x00000000
#define _REG_NNSYS_START 0x00080000
#define _EMUREG_START 0x0000400000
#define _REG_SIZE 0x00080000
#define _REG_NNA_SIZE 0x00080000
#define _REG_NNSYS_SIZE 0x00080000
#define _EMUREG_SIZE 0x0000104000

5471
drivers/nna/include/hwdefs/vha_cr_aura.h Normal file
View File

File diff suppressed because it is too large Load Diff

4998
drivers/nna/include/hwdefs/vha_cr_gyrus.h Normal file
View File

File diff suppressed because it is too large Load Diff

6553
drivers/nna/include/hwdefs/vha_cr_magna.h Normal file
View File

File diff suppressed because it is too large Load Diff

3171
drivers/nna/include/hwdefs/vha_cr_mirage.h Normal file
View File

File diff suppressed because it is too large Load Diff

101
drivers/nna/include/hwdefs/vha_tb.h Normal file
View File

@@ -0,0 +1,101 @@
/*************************************************************************/ /*!
@Title Hardware definition file vha_tb.h
@Copyright Copyright (c) Imagination Technologies Ltd. All Rights Reserved
*/ /**************************************************************************/
/* **** Autogenerated C -- do not edit **** */
/*
*/
#ifndef _VHA_TB_H_
#define _VHA_TB_H_
#define VHA_TB_REVISION 1
/*
Register VHA_TB_MEM_CTRL
*/
#define VHA_TB_MEM_CTRL (0x0000U)
#define VHA_TB_MEM_CTRL_MASKFULL (IMG_UINT64_C(0x00000000FF800000))
#define VHA_TB_MEM_CTRL_MEM_READ_OUTSTANDING_SHIFT (24U)
#define VHA_TB_MEM_CTRL_MEM_READ_OUTSTANDING_CLRMSK (IMG_UINT64_C(0XFFFFFFFF00FFFFFF))
#define VHA_TB_MEM_CTRL_MEM_READ_8TIMES_OUT_SHIFT (23U)
#define VHA_TB_MEM_CTRL_MEM_READ_8TIMES_OUT_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFF7FFFFF))
#define VHA_TB_MEM_CTRL_MEM_READ_8TIMES_OUT_EN (IMG_UINT64_C(0X0000000000800000))
/*
Register VHA_TB_MEM_CTRL_EXT
*/
#define VHA_TB_MEM_CTRL_EXT (0x0010U)
#define VHA_TB_MEM_CTRL_EXT_MASKFULL (IMG_UINT64_C(0x0000000000170000))
#define VHA_TB_MEM_CTRL_EXT_MEM_ENCRYPT_SHIFT (20U)
#define VHA_TB_MEM_CTRL_EXT_MEM_ENCRYPT_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFEFFFFF))
#define VHA_TB_MEM_CTRL_EXT_MEM_ENCRYPT_EN (IMG_UINT64_C(0X0000000000100000))
#define VHA_TB_MEM_CTRL_EXT_MEM_RDATA_INTERLEAVED_MODE_SHIFT (18U)
#define VHA_TB_MEM_CTRL_EXT_MEM_RDATA_INTERLEAVED_MODE_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFBFFFF))
#define VHA_TB_MEM_CTRL_EXT_MEM_RDATA_INTERLEAVED_MODE_EN (IMG_UINT64_C(0X0000000000040000))
#define VHA_TB_MEM_CTRL_EXT_MEM_WRESP_REORDER_MODE_SHIFT (17U)
#define VHA_TB_MEM_CTRL_EXT_MEM_WRESP_REORDER_MODE_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFDFFFF))
#define VHA_TB_MEM_CTRL_EXT_MEM_WRESP_REORDER_MODE_EN (IMG_UINT64_C(0X0000000000020000))
#define VHA_TB_MEM_CTRL_EXT_MEM_RDATA_REORDER_MODE_SHIFT (16U)
#define VHA_TB_MEM_CTRL_EXT_MEM_RDATA_REORDER_MODE_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFEFFFF))
#define VHA_TB_MEM_CTRL_EXT_MEM_RDATA_REORDER_MODE_EN (IMG_UINT64_C(0X0000000000010000))
/*
Register VHA_TB_VHA_IDLE_STATUS
*/
#define VHA_TB_VHA_IDLE_STATUS (0x0060U)
#define VHA_TB_VHA_IDLE_STATUS_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define VHA_TB_VHA_IDLE_STATUS_VHA_IDLE_SHIFT (0U)
#define VHA_TB_VHA_IDLE_STATUS_VHA_IDLE_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFFFFE))
#define VHA_TB_VHA_IDLE_STATUS_VHA_IDLE_EN (IMG_UINT64_C(0X0000000000000001))
/*
Register VHA_TB_VHA_IRQ_STATUS
*/
#define VHA_TB_VHA_IRQ_STATUS (0x0068U)
#define VHA_TB_VHA_IRQ_STATUS_MASKFULL (IMG_UINT64_C(0x0000000000000001))
#define VHA_TB_VHA_IRQ_STATUS_VHA_IRQ_SHIFT (0U)
#define VHA_TB_VHA_IRQ_STATUS_VHA_IRQ_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFFFFE))
#define VHA_TB_VHA_IRQ_STATUS_VHA_IRQ_EN (IMG_UINT64_C(0X0000000000000001))
/*
Register VHA_TB_MEM_WR_CTRL
*/
#define VHA_TB_MEM_WR_CTRL (0x0128U)
#define VHA_TB_MEM_WR_CTRL_MASKFULL (IMG_UINT64_C(0x00000000BFFF3FFF))
#define VHA_TB_MEM_WR_CTRL_MEM_WR_LATENCY_EN_SHIFT (31U)
#define VHA_TB_MEM_WR_CTRL_MEM_WR_LATENCY_EN_CLRMSK (IMG_UINT64_C(0XFFFFFFFF7FFFFFFF))
#define VHA_TB_MEM_WR_CTRL_MEM_WR_LATENCY_EN_EN (IMG_UINT64_C(0X0000000080000000))
#define VHA_TB_MEM_WR_CTRL_MEM_WR_LATENCY_MAX_SHIFT (16U)
#define VHA_TB_MEM_WR_CTRL_MEM_WR_LATENCY_MAX_CLRMSK (IMG_UINT64_C(0XFFFFFFFFC000FFFF))
#define VHA_TB_MEM_WR_CTRL_MEM_WR_LATENCY_MIN_SHIFT (0U)
#define VHA_TB_MEM_WR_CTRL_MEM_WR_LATENCY_MIN_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFC000))
/*
Register VHA_TB_MEM_RD_CTRL
*/
#define VHA_TB_MEM_RD_CTRL (0x0130U)
#define VHA_TB_MEM_RD_CTRL_MASKFULL (IMG_UINT64_C(0x00000000BFFF3FFF))
#define VHA_TB_MEM_RD_CTRL_MEM_RD_LATENCY_EN_SHIFT (31U)
#define VHA_TB_MEM_RD_CTRL_MEM_RD_LATENCY_EN_CLRMSK (IMG_UINT64_C(0XFFFFFFFF7FFFFFFF))
#define VHA_TB_MEM_RD_CTRL_MEM_RD_LATENCY_EN_EN (IMG_UINT64_C(0X0000000080000000))
#define VHA_TB_MEM_RD_CTRL_MEM_RD_LATENCY_MAX_SHIFT (16U)
#define VHA_TB_MEM_RD_CTRL_MEM_RD_LATENCY_MAX_CLRMSK (IMG_UINT64_C(0XFFFFFFFFC000FFFF))
#define VHA_TB_MEM_RD_CTRL_MEM_RD_LATENCY_MIN_SHIFT (0U)
#define VHA_TB_MEM_RD_CTRL_MEM_RD_LATENCY_MIN_CLRMSK (IMG_UINT64_C(0XFFFFFFFFFFFFC000))
#endif /* _VHA_TB_H_ */
/*****************************************************************************
End of file (vha_tb.h)
*****************************************************************************/

296
drivers/nna/include/img_mem_man.h Normal file
View File

@@ -0,0 +1,296 @@
/*!
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef IMG_MEM_MAN_H
#define IMG_MEM_MAN_H
#include <linux/version.h>
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,14,0)
#define KERNEL_DMA_FENCE_SUPPORT
#endif
#include <linux/mm.h>
#include <linux/types.h>
#include <linux/device.h>
#ifdef KERNEL_DMA_FENCE_SUPPORT
#include <linux/dma-fence.h>
#endif
#include "uapi/img_mem_man.h"
/* Defines allocation order range for platforms
* that do not explicitly defined it.
* NOTE: applicable for unified heap type only */
#define IMG_MIN_ALLOC_ORDER_DEFAULT 0
#define IMG_MAX_ALLOC_ORDER_DEFAULT 0
/* Page catalogue address shift */
#define IMG_MMU_PC_ADDR_SHIFT 12
/* MMUv3 PTE entry flags */
enum {
IMG_MMU_PTE_FLAG_NONE = 0x0,
IMG_MMU_PTE_FLAG_VALID = 0x1,
IMG_MMU_PTE_FLAG_READ_ONLY = 0x2,
IMG_MMU_PTE_FLAG_CACHE_COHERENCY = 0x4,
};
/* All level entries flags are stored under 4lsb bits */
#define IMG_MMU_ENTRY_FLAGS_MASK 0xf
/* Each entry can store 40bit physical address */
#define IMG_MMU_PHY_ADDR_MASK ((1ULL<<40)-1)
struct mmu_config {
uint32_t addr_width; /* physical */
bool bypass_hw; /* MMU bypass mode */
size_t bypass_offset; /* Optional offset in physical space for MMU bypass mode */
bool use_pte_parity; /* enables parity calculation for PTEs */
/* memory attributes to be used when allocating mmu pages */
enum img_mem_attr alloc_attr;
int page_size;
};
union heap_options {
struct {
gfp_t gfp_type; /* pool and flags for buffer allocations */
int min_order; /* minimum page allocation order */
int max_order; /* maximum page allocation order */
} unified;
#ifdef CONFIG_ION
struct {
struct ion_client *client; /* must be provided by platform */
} ion;
#endif
#ifdef CONFIG_GENERIC_ALLOCATOR
struct {
void *kptr; /* static pointer to kernel mapping of memory */
/* Optional hooks to obtain kernel mapping dynamically */
void* (*get_kptr)(
phys_addr_t addr,
size_t size,
enum img_mem_attr mattr);
int (*put_kptr)(void *);
phys_addr_t phys; /* physical address start of memory */
size_t size; /* size of memory */
unsigned long offs; /* optional offset of the start
of memory as seen from device,
zero by default */
int pool_order; /* allocation order */
} carveout;
#endif
struct {
bool use_sg_dma; /* Forces sg_dma physical address instead of CPU physical address*/
} dmabuf;
struct {
gfp_t gfp_flags; /* for buffer allocations */
} coherent;
struct {
phys_addr_t phys; /* physical address start of memory */
size_t size; /* size of memory */
enum img_mem_heap_attrs hattr; /* User attributes */
} ocm;
};
struct heap_config {
enum img_mem_heap_type type;
union heap_options options;
/* (optional) functions to convert a physical address as seen from
the CPU to the physical address as seen from the vha device and
vice versa. When not implemented,
it is assumed that physical addresses are the
same regardless of viewpoint */
phys_addr_t (*to_dev_addr)(union heap_options *opts, phys_addr_t addr);
phys_addr_t (*to_host_addr)(union heap_options *opts, phys_addr_t addr);
/* Cache attribute,
* could be platform specific if provided - overwrites the global cache policy */
enum img_mem_attr cache_attr;
};
enum img_mmu_callback_type {
IMG_MMU_CALLBACK_MAP = 1,
IMG_MMU_CALLBACK_UNMAP,
};
struct mem_ctx;
struct mmu_ctx;
int img_mem_add_heap(const struct heap_config *heap_cfg, int *heap_id);
void img_mem_del_heap(int heap_id);
int img_mem_get_heap_info(int heap_id, uint8_t *type, uint32_t *attrs);
/*
* related to process context (contains SYSMEM heap's functionality in general)
*/
int img_mem_create_proc_ctx(struct mem_ctx **ctx);
void img_mem_destroy_proc_ctx(struct mem_ctx *ctx);
int img_mem_alloc(struct device *device, struct mem_ctx *ctx, int heap_id,
size_t size, enum img_mem_attr attributes, int *buf_id);
int img_mem_import(struct device *device, struct mem_ctx *ctx, int heap_id,
size_t size, enum img_mem_attr attributes, uint64_t buf_hnd,
int *buf_id);
int img_mem_export(struct device *device, struct mem_ctx *ctx, int buf_id,
size_t size, enum img_mem_attr attributes, uint64_t *buf_hnd);
void img_mem_free(struct mem_ctx *ctx, int buf_id);
int img_mem_map_um(struct mem_ctx *ctx, int buf_id, struct vm_area_struct *vma);
int img_mem_unmap_um(struct mem_ctx *ctx, int buf_id);
int img_mem_map_km(struct mem_ctx *ctx, int buf_id);
int img_mem_unmap_km(struct mem_ctx *ctx, int buf_id);
void *img_mem_get_kptr(struct mem_ctx *ctx, int buf_id);
uint64_t *img_mem_get_page_array(struct mem_ctx *mem_ctx, int buf_id);
uint64_t img_mem_get_single_page(struct mem_ctx *mem_ctx, int buf_id,
unsigned int offset);
phys_addr_t img_mem_get_dev_addr(struct mem_ctx *mem_ctx, int buf_id,
phys_addr_t addr);
int img_mem_sync_cpu_to_device(struct mem_ctx *ctx, int buf_id);
int img_mem_sync_device_to_cpu(struct mem_ctx *ctx, int buf_id);
int img_mem_get_usage(const struct mem_ctx *ctx, size_t *max, size_t *curr);
int img_mmu_get_usage(const struct mem_ctx *ctx, size_t *max, size_t *curr);
#ifdef KERNEL_DMA_FENCE_SUPPORT
struct dma_fence * img_mem_add_fence(struct mem_ctx *ctx, int buf_id);
void img_mem_remove_fence(struct mem_ctx *ctx, int buf_id);
int img_mem_signal_fence(struct mem_ctx *ctx, int buf_id);
#endif
/*
* related to stream MMU context (constains IMGMMU functionality in general)
*/
int img_mmu_ctx_create(struct device *device, const struct mmu_config *config,
struct mem_ctx *mem_ctx, int heap_id,
int (*callback_fn)(enum img_mmu_callback_type type,
int buf_id, void *data),
void *callback_data,
struct mmu_ctx **mmu_ctx);
void img_mmu_ctx_destroy(struct mmu_ctx *mmu);
int img_mmu_map(struct mmu_ctx *mmu_ctx, struct mem_ctx *mem_ctx, int buf_id,
uint64_t virt_addr, unsigned int map_flags);
int img_mmu_unmap(struct mmu_ctx *mmu_ctx, struct mem_ctx *mem_ctx, int buf_id);
int img_mmu_get_pc(const struct mmu_ctx *ctx,
unsigned int *pc_reg, int *buf_id);
int img_mmu_get_conf(size_t *page_size, size_t *virt_size);
phys_addr_t img_mmu_get_paddr(const struct mmu_ctx *ctx,
uint64_t vaddr, uint8_t *flags);
int img_mmu_init_cache(struct mmu_ctx *mmu_ctx, unsigned long cache_phys_start,
uint32_t cache_size);
int img_mmu_clear_cache(struct mmu_ctx *mmu_ctx);
int img_mmu_move_pg_to_cache(struct mmu_ctx *mmu_ctx, struct mem_ctx *mem_ctx,
int buf_id, uint64_t virt_addr, uint32_t page_size, uint32_t page_idx);
/*
* virtual address allocation
*/
struct mmu_vaa;
int img_mmu_vaa_create(struct device *device,
uint32_t base, size_t size, struct mmu_vaa **vaa);
int img_mmu_vaa_destroy(struct mmu_vaa *vaa);
int img_mmu_vaa_alloc(struct mmu_vaa *vaa, size_t size, uint32_t *addr);
int img_mmu_vaa_free(struct mmu_vaa *vaa, uint32_t addr, size_t size);
bool img_mem_calc_parity(unsigned long long input);
/*
* PDUMP generation:
* img_pdump_txt_create creates a TXT buffer in RAM,
* which is used by img_pdump_txt_printf
* img_pdump_bin_create creates a PRM or RES buffer in RAM,
* which is used by img_pdump_bin_write
*
*/
struct pdump_buf {
char *ptr;
size_t size; /* allocated size of buffer */
size_t len; /* how full is the buffer */
bool drop_data; /* do not store data in file */
};
#define PDUMP_TXT 0 /* eg pdump.txt */
#define PDUMP_PRM 1 /* eg pdump.prm */
#define PDUMP_RES 2 /* eg pdump.res */
#define PDUMP_DBG 3 /* eg pdump.dbg */
#define PDUMP_CRC 4 /* eg pdump.crc */
#define PDUMP_CRC_CMB 5 /* eg pdump.crc_cmb */
#define PDUMP_MAX 6
/*
* VHA PDUMPs.
* Uses img_pdump buffers to collect pdump information.
* there are 3 different PDUMP files: TXT, PRM and RES.
* they are simply buffers in ram.
* They are mapped into debugfs: /sys/kernel/debug/vhaN/pdump.*
*/
struct pdump_descr {
struct pdump_buf pbufs[PDUMP_MAX];
struct mutex lock;
};
#ifndef OSID
#define _PMEM_ ":MEM"
#else
#define _PMEM_ ":MEM_OS"__stringify(OSID)
#endif
struct pdump_buf *img_pdump_create(struct pdump_descr* pdump, uint32_t pdump_num, size_t size);
int img_pdump_write(struct pdump_descr* pdump, uint32_t pdump_num, const void *ptr, size_t size);
int __img_pdump_printf(struct device* dev, const char *fmt, ...) __printf(2, 3);
#define img_pdump_printf(fmt, ...) __img_pdump_printf(vha->dev, fmt, ##__VA_ARGS__)
void img_pdump_destroy(struct pdump_descr* pdump);
bool img_pdump_enabled(struct pdump_descr* pdump);
#endif /* IMG_MEM_MAN_H */
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

55
drivers/nna/include/nexef_plat.h Normal file
View File

@@ -0,0 +1,55 @@
/*!
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef GYRUS_PLAT_H
#define GYRUS_PLAT_H
#define NEXEF_NNA_DEVICE_NAME "nexef_nna"
struct nexef_nna_platform_data {
uint64_t nna_memory_offset;
uint64_t nna_memory_base;
uint64_t nna_memory_size;
};
#define TC_INTERRUPT_NNA 2
#endif /* NEXEF_PLAT_H */

118
drivers/nna/include/uapi/img_mem_man.h Normal file
View File

@@ -0,0 +1,118 @@
/*!
*****************************************************************************
*
* @File img_mem_man.h
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef IMG_MEM_MAN_UAPI_H
#define IMG_MEM_MAN_UAPI_H
/* memory attributes */
enum img_mem_attr {
IMG_MEM_ATTR_CACHED = 0x00000001,
IMG_MEM_ATTR_UNCACHED = 0x00000002,
IMG_MEM_ATTR_WRITECOMBINE = 0x00000004,
/* Special */
IMG_MEM_ATTR_SECURE = 0x00000010,
IMG_MEM_ATTR_NOMAP = 0x00000020,
IMG_MEM_ATTR_NOSYNC = 0x00000040,
/* Internal */
IMG_MEM_ATTR_MMU = 0x10000000,
IMG_MEM_ATTR_OCM = 0x20000000,
};
/* Cache attributes mask */
#define IMG_MEM_ATTR_CACHE_MASK 0xf
/* Supported heap types */
enum img_mem_heap_type {
IMG_MEM_HEAP_TYPE_UNKNOWN = 0,
IMG_MEM_HEAP_TYPE_UNIFIED,
IMG_MEM_HEAP_TYPE_CARVEOUT,
IMG_MEM_HEAP_TYPE_ION,
IMG_MEM_HEAP_TYPE_DMABUF,
IMG_MEM_HEAP_TYPE_COHERENT,
IMG_MEM_HEAP_TYPE_ANONYMOUS,
IMG_MEM_HEAP_TYPE_OCM,
};
/* Heap attributes */
enum img_mem_heap_attrs {
IMG_MEM_HEAP_ATTR_INTERNAL = 0x01,
IMG_MEM_HEAP_ATTR_IMPORT = 0x02,
IMG_MEM_HEAP_ATTR_EXPORT = 0x04,
IMG_MEM_HEAP_ATTR_SEALED = 0x08,
/* User attributes */
IMG_MEM_HEAP_ATTR_LOCAL = 0x10,
IMG_MEM_HEAP_ATTR_SHARED = 0x20,
};
/* heaps ids */
#define IMG_MEM_MAN_HEAP_ID_INVALID 0
#define IMG_MEM_MAN_MIN_HEAP 1
#define IMG_MEM_MAN_MAX_HEAP 16
/* buffer ids (per memory context) */
#define IMG_MEM_MAN_BUF_ID_INVALID 0
#define IMG_MEM_MAN_MIN_BUFFER 1
#define IMG_MEM_MAN_MAX_BUFFER 2000
/* Definition of VA guard gap between allocations */
#define IMG_MEM_VA_GUARD_GAP 0x1000
/* Virtual memory space for buffers allocated
* in the kernel - OCM & device debug buffers */
#define IMG_MEM_VA_HEAP1_BASE 0x8000000ULL
#define IMG_MEM_VA_HEAP1_SIZE 0x40000000ULL
/* Definition of VA guard gap between heaps - 2MB (size of MMU PD) */
#define IMG_MEM_HEAP_GUARD_GAP 0x200000
/* Virtual memory space for buffers allocated in the user space */
#define IMG_MEM_VA_HEAP2_BASE ( \
IMG_MEM_VA_HEAP1_BASE + IMG_MEM_VA_HEAP1_SIZE + IMG_MEM_HEAP_GUARD_GAP)
#define IMG_MEM_VA_HEAP2_SIZE 0x3C0000000ULL
#endif /* IMG_MEM_MAN_UAPI_H */

49
drivers/nna/include/uapi/version.h Normal file
View File

@@ -0,0 +1,49 @@
/*!
******************************************************************************
@file : version.h
@brief Version information for VHA tools
@Author Imagination Technologies
@date 08/05/2013
@License MIT
Copyright (c) Imagination Technologies Ltd.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
\n<b>Description:</b>\n
This file is automatically updated by the build system to contain
the correct version information.
\n<b>Platform:</b>\n
Platform Independent
******************************************************************************/
#ifndef VERSION_H
#define VERSION_H
#define VERSION_STRING "REL_3.8-cl6140200"
#define KERNEL_INTERFACE_DIGEST "37b909e8b218177ddfc5eb2d5f162348"
#endif // VERSION_H

423
drivers/nna/include/uapi/vha.h Normal file
View File

@@ -0,0 +1,423 @@
/*!
*****************************************************************************
*
* @File vha.h
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef _VHA_UAPI_H
#define _VHA_UAPI_H
#if defined(__KERNEL__)
#include <linux/ioctl.h>
#include <linux/types.h>
#elif defined(__linux__)
#include <sys/ioctl.h>
#include <inttypes.h>
#else
#error unsupported build
#endif
#if defined(__cplusplus)
extern "C" {
#endif
#define VHA_OCM_MAX_NUM_PAGES 128
#define VHA_CORE_MAX_ALT_ADDRS 16
#define VHA_MAX_CORES 8
// represents OCM types,
#define VHA_LOCAL_OCM 0 /* Local OCM */
#define VHA_SHARED_OCM 1 /* Shared OCM */
#define VHA_OCM_TYPE_MAX 2
/* device hw properties */
struct vha_hw_props {
uint64_t product_id;
uint64_t core_id;
uint64_t soc_axi;
uint8_t mmu_width; /* MMU address width: 40, or 0 if no MMU */
uint8_t mmu_ver; /* MMU version */
uint32_t mmu_pagesize; /* MMU page size */
union {
struct {
unsigned rtm: 1;
unsigned parity: 1;
} supported;
uint8_t features;
};
bool dummy_dev;
bool skip_bvnc_check;
bool use_pdump;
uint8_t num_cnn_core_devs;
uint32_t locm_size_bytes; /* per core */
uint32_t socm_size_bytes; /* total size for all cores */
uint32_t socm_core_size_bytes; /* per core */
uint32_t clock_freq; /* hardware clock rate, kHz */
} __attribute__((aligned(8)));
struct vha_cnn_props {
/* TOBEDONE */
};
/* command sent to device */
enum vha_cmd_type {
VHA_CMD_INVALID = 0x000,
VHA_CMD_CNN_SUBMIT = 0x101,
VHA_CMD_CNN_SUBMIT_MULTI,
VHA_CMD_CNN_PDUMP_MSG
};
/* optional flags for commands */
#define VHA_CMDFLAG_NOTIFY 0x0001 /* send response when cmd complete */
#define VHA_CHECK_CRC 0x0002 /* check the combined CRCs */
#define VHA_EXEC_TIME_SET 0x0004 /* execution time is valid */
/*
* message from user to be sent to VHA (write).
* A command will contain a number of input and ouput buffers
* and some command specific parameters.
* Buffers must be identified by their buffer id.
* All buffer ids must *precede* any other parameters:
* input buf ids,
* output buf ids,
* followed by other parameters.
*/
struct vha_user_cmd {
uint32_t cmd_id; /* arbitrary id for cmd */
uint16_t cmd_type; /* enum vha_cmd_type */
uint16_t flags; /* VHA_CMDFLAG_xxx */
uint8_t priority; /* WL priority */
uint8_t padding; /* padding to keep data 32bit aligned */
uint8_t num_bufs; /* total number of buffers */
uint8_t num_inbufs; /* number of input buffers */
uint32_t data[0]; /* 0-N words: input bufids
* followed by other bufids
* followed by other parameters */
};
/* Structure defining hardware issues */
struct vha_hw_brns {
union {
struct {
unsigned bRN71649: 1;
unsigned bRN71556: 1;
unsigned bRN71338: 1;
} bit;
uint64_t map;
};
};
/*
* CNN_SUBMIT message written from user to VHA.
* 3 input buffers: cmdstream, input image, coefficients
* 1 output buffer
* 1 internal buffer (optional)
* offsets into the input and output buffers
* and a register map: this tells the driver which alt-register-N
* will contain the address of which buffer.
*/
struct vha_subseg_info {
uint32_t cmdbuf_offset;
uint32_t cmdbuf_size;
};
struct vha_user_cnn_submit_cmd {
struct vha_user_cmd msg;
uint32_t cmdbuf; /* bufid of cmdstream buffer */
uint32_t bufs[VHA_CORE_MAX_ALT_ADDRS]; /* bufid of IN, COEFF, OUT, INTERNAL,CRC,DBG,WB buffers */
uint32_t bufoffsets[VHA_CORE_MAX_ALT_ADDRS]; /* offsets into inbufs and outbufs buffers */
uint32_t bufsizes[VHA_CORE_MAX_ALT_ADDRS]; /* sizes of the inbufs and outbufs buffers */
uint8_t regidx[VHA_CORE_MAX_ALT_ADDRS]; /* register to be used for inbufs and outbufs */
uint32_t onchipram_map_id; /* OCM mapping id - hot pages */
uint32_t onchipram_bufs[VHA_OCM_TYPE_MAX]; /* OCM linear mapping buffers */
uint32_t estimated_cycles; /* estimated number of cycles for this command */
uint64_t expected_ip_capab; /* expected BVNC */
uint64_t hw_brns; /* BRNSs bit map */
uint32_t subseg_num; /* number of subsegments in subseg_info array */
struct vha_subseg_info subseg_info[1]; /* there's always at least one subsegment */
} __attribute__((aligned(8)));
/*
* CNN_SUBMIT_MULTI message written from user to VHA.
* 3 input buffers: cmdstream(s), input image, coefficients
* 1 output buffer
* 1 internal buffer (optional)
* offsets into the input and output buffers
* and a register map: this tells the driver which alt-register-N
* will contain the address of which buffer.
*/
struct vha_user_cnn_submit_multi_cmd {
struct vha_user_cmd msg;
uint32_t cmdbuf[VHA_MAX_CORES]; /* bufid of cmdstream buffer */
uint32_t bufs[VHA_CORE_MAX_ALT_ADDRS]; /* bufid of IN, COEFF, OUT, INTERNAL,CRC,DBG,WB buffers */
uint32_t bufoffsets[VHA_CORE_MAX_ALT_ADDRS]; /* offsets into inbufs and outbufs buffers */
uint32_t bufsizes[VHA_CORE_MAX_ALT_ADDRS]; /* sizes of the inbufs and outbufs buffers */
uint8_t regidx[VHA_CORE_MAX_ALT_ADDRS]; /* register to be used for inbufs and outbufs */
uint8_t num_cores; /* number of cores required for this workload */
uint32_t onchipram_bufs[VHA_OCM_TYPE_MAX]; /* OCM linear mapping buffers */
uint32_t crcs[VHA_MAX_CORES]; /* golden CRCs */
uint64_t exec_time; /* expected execution time */
uint32_t shared_circ_buf_offs; /* circular buffer offset in the shared memory */
uint32_t estimated_cycles; /* estimated number of cycles for this command */
uint64_t expected_ip_capab; /* expected BVNC */
uint64_t hw_brns; /* BRNSs bit map */
} __attribute__((aligned(8)));
/*
* response from from kernel module to user.
*/
struct vha_user_rsp {
uint32_t cmd_id; /* arbitrary id to identify cmd */
uint32_t err_no; /* 0 if successful, else -ve */
uint64_t rsp_err_flags;
uint32_t data[0]; /* 0-N words of additional info */
};
/*
* response returned after CNN_SUBMIT.
*/
struct vha_user_cnn_submit_rsp {
struct vha_user_rsp msg;
uint64_t last_proc_us; /* processing time in us,
measured with system clock */
uint32_t mem_usage; /* device memory used */
uint32_t hw_cycles; /* hardware cycles used */
} __attribute__((aligned(8)));
#define MAX_VHA_USER_RSP_SIZE (sizeof(struct vha_user_cnn_submit_rsp))
/* response returned when querying for heaps */
struct vha_heap_data {
uint32_t id; /* Heap ID */
uint32_t type; /* Heap type */
uint32_t attributes; /* Heap attributes
defining capabilities that user may treat as hint
when selecting the heap id during allocation/importing */
};
#define VHA_MAX_HEAPS 16
struct vha_heaps_data {
struct vha_heap_data heaps[VHA_MAX_HEAPS]; /* [OUT] Heap data */
} __attribute__((aligned(8)));
/* parameters to allocate a device buffer */
struct vha_alloc_data {
uint64_t size; /* [IN] Size of device memory (in bytes) */
uint32_t heap_id; /* [IN] Heap ID of allocator
or VHA_USE_DEFAULT_MEM_HEAP */
uint32_t attributes; /* [IN] Attributes of buffer: img_mem_attr */
char name[8]; /* [IN] short name for buffer */
uint32_t buf_id; /* [OUT] Generated buffer ID */
} __attribute__((aligned(8)));
/* parameters to import a device buffer */
struct vha_import_data {
uint64_t size; /* [IN] Size of device memory (in bytes) */
uint64_t buf_hnd; /* [IN] File descriptor/cpu pointer
of buffer to import */
uint32_t heap_id; /* [IN] Heap ID of allocator */
uint32_t attributes; /* [IN] Attributes of buffer */
char name[8]; /* [IN] short name for buffer */
uint32_t buf_id; /* [OUT] Generated buffer ID */
} __attribute__((aligned(8)));
/* parameters to export a device buffer */
struct vha_export_data {
uint32_t buf_id; /* [IN] Buffer ID to be exported */
uint64_t size; /* [IN] Size to be exported */
uint32_t attributes; /* [IN] Attributes of buffer */
uint64_t buf_hnd; /* [OUT] Buffer handle (file descriptor) */
} __attribute__((aligned(8)));
struct vha_free_data {
uint32_t buf_id; /* [IN] ID of device buffer to free */
};
enum vha_map_flags {
VHA_MAP_FLAG_NONE = 0x0,
VHA_MAP_FLAG_READ_ONLY = 0x1,
VHA_MAP_FLAG_WRITE_ONLY = 0x2,
VHA_MAP_FLAG_IO = 0x4,
VHA_MAP_FLAG_MODEL = 0x8,
};
/* parameters to map a buffer into device */
struct vha_map_to_onchip_data {
uint64_t virt_addr; /* [IN] Device virtual address of a mapping */
uint32_t buf_id; /* [IN] ID of device buffer to map to VHA */
uint32_t page_size; /* [IN] Page size */
uint32_t num_pages; /* [IN] The number of pages to be mapped */
uint32_t page_idxs[VHA_OCM_MAX_NUM_PAGES];
/* [IN] Indexes of pages to be mapped */
uint32_t map_id; /* [IN/OUT] if map_id == 0, creates new mapping
and generates new map_id,
otherwise using existing map_id*/
} __attribute__((aligned(8)));
/* parameters to map a buffer into device */
struct vha_map_data {
uint64_t virt_addr; /* [IN] Device virtual address to map */
uint32_t buf_id; /* [IN] ID of device buffer to map to VHA */
uint32_t flags; /* [IN] Mapping flags, see vha_map_flags */
} __attribute__((aligned(8)));
struct vha_unmap_data {
uint32_t buf_id; /* [IN] ID of device buffer to unmap from VHA */
} __attribute__((aligned(8)));
enum vha_buf_status {
VHA_BUF_UNFILLED,
VHA_BUF_FILLED_BY_SW,
VHA_BUF_FILLED_BY_HW
};
#define VHA_SYNC_NONE (-1)
/* parameters to set buffer status ("filled" or "unfilled") */
struct vha_buf_status_data {
uint32_t buf_id;
uint32_t status; /* enum vha_buf_status */
int in_sync_fd; /* input sync to attach */
bool out_sync_sig; /* output sync signal */
} __attribute__((aligned(8)));
enum vha_sync_op {
VHA_SYNC_OP_CREATE_OUT, /* create output sync_fd */
VHA_SYNC_OP_MERGE_IN, /* merge input sync_fds */
VHA_SYNC_OP_RELEASE /* release syncs */
};
/* parameters to manage sync_fds */
#define VHA_SYNC_MAX_BUF_IDS (VHA_CORE_MAX_ALT_ADDRS)
#define VHA_SYNC_MAX_IN_SYNC_FDS (VHA_CORE_MAX_ALT_ADDRS)
struct vha_sync_create_data {
uint32_t buf_id_count; /* [IN] number of output buffers */
uint32_t buf_ids[VHA_SYNC_MAX_BUF_IDS]; /* [IN] list of output buffer ids */
};
struct vha_sync_merge_data {
uint32_t in_sync_fd_count; /* [IN] number of input sync_fds */
int in_sync_fds[VHA_SYNC_MAX_IN_SYNC_FDS]; /* [IN] list of input sync_fds */
};
struct vha_sync_release_data {
uint32_t buf_id_count; /* [IN] number of buffers */
uint32_t buf_ids[VHA_SYNC_MAX_BUF_IDS]; /* [IN] list of buffer ids */
};
struct vha_sync_data {
enum vha_sync_op op;
union {
struct vha_sync_create_data create_data; /* create output sync_fd data */
struct vha_sync_merge_data merge_data; /* merge input sync_fds data */
struct vha_sync_release_data release_data; /* release syncs data */
};
int sync_fd; /* [OUT] output sync_fd/sync_fd for merged input sync_fds */
} __attribute__((aligned(8)));
struct vha_cancel_data {
uint32_t cmd_id; /* [IN] masked ID of commands to be cancelled */
uint32_t cmd_id_mask; /* [IN] mask for command IDs to be cancelled */
bool respond; /* [IN] if true, respond to this cancel request */
} __attribute__((aligned(8)));
struct vha_version_data {
char digest[33]; /* [OUT] digest of this interface file */
} __attribute__((aligned(8)));
#define VHA_IOC_MAGIC 'q'
#define VHA_IOC_HW_PROPS _IOR(VHA_IOC_MAGIC, 0, struct vha_hw_props)
#define VHA_IOC_QUERY_HEAPS _IOR(VHA_IOC_MAGIC, 1, struct vha_heaps_data)
#define VHA_IOC_ALLOC _IOWR(VHA_IOC_MAGIC, 2, struct vha_alloc_data)
#define VHA_IOC_IMPORT _IOWR(VHA_IOC_MAGIC, 3, struct vha_import_data)
#define VHA_IOC_EXPORT _IOWR(VHA_IOC_MAGIC, 4, struct vha_export_data)
#define VHA_IOC_FREE _IOW(VHA_IOC_MAGIC, 5, struct vha_free_data)
#define VHA_IOC_VHA_MAP_TO_ONCHIP _IOW(VHA_IOC_MAGIC, 6, struct vha_map_to_onchip_data)
#define VHA_IOC_VHA_MAP _IOW(VHA_IOC_MAGIC, 7, struct vha_map_data)
#define VHA_IOC_VHA_UNMAP _IOW(VHA_IOC_MAGIC, 8, struct vha_unmap_data)
#define VHA_IOC_BUF_STATUS _IOW(VHA_IOC_MAGIC, 9, struct vha_buf_status_data)
#define VHA_IOC_SYNC _IOWR(VHA_IOC_MAGIC, 10, struct vha_sync_data)
#define VHA_IOC_CANCEL _IOW(VHA_IOC_MAGIC, 11, struct vha_cancel_data)
#define VHA_IOC_VERSION _IOW(VHA_IOC_MAGIC, 16, struct vha_version_data)
#define VHA_SCOPE_DEV_NAME "vha_scope"
/* vha scope context
* */
struct vha_trace_ctx {
unsigned model_id; /* model id */
unsigned frame_id; /* inference id */
unsigned dev_id; /* device id */
unsigned osid; /* device id */
unsigned pid; /* process id */
unsigned tid; /* thread id */
};
/* Event information, available from vha_info */
struct vha_timing_data {
unsigned evt_type; /* event type */
unsigned seqno; /* continually increments */
unsigned dev_id; /* device id */
unsigned timestamp_lo; /* in microseconds */
unsigned timestamp_hi;
unsigned type; /* either SUBMIT or COMPLETE or ERROR */
unsigned cycles; /* HW cycle count */
unsigned pid; /* process id */
};
enum vha_scope_evt_type {
VHA_EVENT_TIMING,
VHA_EVENT_NUM
};
enum vha_timing_data_type {
VHA_EVENT_TYPE_ENQUEUE,
VHA_EVENT_TYPE_SUBMIT,
VHA_EVENT_TYPE_COMPLETE,
VHA_EVENT_TYPE_ERROR,
VHA_EVENT_TYPE_NUM
};
#if defined(__cplusplus)
}
#endif
#endif /* _VHA_UAPI_H */

116
drivers/nna/include/uapi/vha_errors.h Normal file
View File

@@ -0,0 +1,116 @@
/*!
*****************************************************************************
*
* @File vha_errors.h
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef _VHA_ERRORS_H
#define _VHA_ERRORS_H
#if defined(__cplusplus)
extern "C" {
#endif
enum {
/* System errors */
VHA_RSP_ERROR_HW_SYS_AXI_ERROR_SHF = 0,
VHA_RSP_ERROR_HW_SYS_MMU_PAGE_FAULT_SHF,
VHA_RSP_ERROR_HW_SYS_SYS_MEM_WDT_SHF,
VHA_RSP_ERROR_HW_SYS_AXI_MEMORY_PARITY_ERROR_SHF,
VHA_RSP_ERROR_HW_SYS_MMU_PARITY_ERROR_SHF,
VHA_RSP_ERROR_HW_SYS_RAM_CORRECTION_SHF,
VHA_RSP_ERROR_HW_SYS_RAM_DETECTION_SHF,
VHA_RSP_ERROR_HW_SYS_LSYNC_INV_REQ_SHF,
VHA_RSP_ERROR_HW_SYS_LOGIC_ERROR_SHF,
VHA_RSP_ERROR_SW_SYS_EVNT_PARITY_ERROR_SHF,
VHA_RSP_ERROR_SW_WDT_EXPIRED_SHF,
/* WM event errors */
VHA_RSP_ERROR_HW_EVNT_WM_WL_WDT_SHF,
VHA_RSP_ERROR_HW_EVNT_WM_WL_IDLE_WDT_SHF,
VHA_RSP_ERROR_HW_EVNT_WM_SOCIF_WDT_SHF,
VHA_RSP_ERROR_HW_EVNT_LOGIC_FAULT_SHF,
VHA_RSP_ERROR_SW_EVNT_WM_PARITY_ERROR_SHF,
/* WM response FIFO errors */
VHA_RSP_ERROR_HW_CORE_IRQ_BEFORE_KICK_SHF,
VHA_RSP_ERROR_HW_INDIRECT_MASK_SET_ERROR_SHF,
VHA_RSP_ERROR_HW_KICK_CORE_ACCESS_ERROR_SHF,
VHA_RSP_ERROR_HW_CNN_CONTROL_START_HIGH_SHF,
VHA_RSP_ERROR_HW_CNN_STATUS_ERROR_SHF,
VHA_RSP_ERROR_HW_INT_CORE_ACCESS_ERROR_SHF,
VHA_RSP_ERROR_HW_CORE_EVENT_ERROR_SHF,
VHA_RSP_ERROR_HW_CORE_EVENT_NOT_CLEARED_SHF,
VHA_RSP_ERROR_HW_CORE_EVENT_IRQ_HIGH_SHF,
VHA_RSP_ERROR_HW_INTERCONNECT_ERROR_SHF,
VHA_RSP_ERROR_SW_WM_PARITY_ERROR_SHF,
VHA_RSP_ERROR_SW_WL_ID_MISMATCH_ERROR_SHF,
VHA_RSP_ERROR_SW_CONF_ERROR_SHF,
VHA_RSP_ERROR_SW_CRC_MISMATCH_ERROR_SHF,
/* CNN core status errors. */
VHA_RSP_ERROR_HW_CORE_LOGIC_ERROR_SHF,
VHA_RSP_ERROR_HW_RAM_CORRECTION_SHF,
VHA_RSP_ERROR_HW_RAM_DETECTION_SHF,
VHA_RSP_ERROR_HW_CORE_SYNC_ERROR_SHF,
VHA_RSP_ERROR_HW_CORE_WDT_SHF,
VHA_RSP_ERROR_HW_CORE_MEM_WDT_SHF,
VHA_RSP_ERROR_HW_CORE_CNN_ERROR_SHF,
/* Interconnect status errors. */
VHA_RSP_ERROR_HW_LOCKSTEP_ERROR_SHF,
VHA_RSP_ERROR_HW_IC_LOGIC_ERROR_SHF,
VHA_RSP_ERROR_HW_SOCIF_READ_MISMATCH_SHF,
VHA_RSP_ERROR_HW_SOCIF_READ_UNRESPONSIVE_SHF,
VHA_RSP_ERROR_SW_IC_PARITY_ERROR_SHF,
/* Workload submit errors. */
VHA_RSP_ERROR_SW_SKIP_CMD_SHF,
VHA_RSP_ERROR_SW_KICK_BIT_READ_BACK_FAILURE_SHF,
VHA_RSP_ERROR_SW_HW_BUSY_SHF,
VHA_RSP_ERROR_SW_INVALID_CMD_INFO_SHF,
VHA_RSP_ERROR_SW_INVALID_CMD_TYPE_SHF,
VHA_RSP_ERROR_SW_MMU_SETUP_FAILURE_SHF
};
#define VHA_RSP_ERROR(err) (1ull << (VHA_RSP_ERROR_##err##_SHF))
#if defined(__cplusplus)
}
#endif
#endif /* _VHA_ERRORS_H */

72
drivers/nna/include/vha_drv_common.h Normal file
View File

@@ -0,0 +1,72 @@
/*!
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef VHA_DRV_COMMON_H
#define VHA_DRV_COMMON_H
#include <img_mem_man.h>
/* This is a structure shared between img_mem and vha modules.
* The pointer to it is stored in device.driver_data
*/
struct vha_dev_common {
void* vha_dev; /* opaque pointer to vha_dev structure */
struct pdump_descr pdump;
};
static inline struct pdump_descr* vha_pdump_dev_get_drvdata(struct device* dev) {
struct vha_dev_common* vdc = dev_get_drvdata(dev);
if(!vdc)
return NULL;
return &vdc->pdump;
}
#endif /* VHA_DRV_COMMON_H */
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

24
drivers/nna/nexef_platform/Makefile Normal file
View File

@@ -0,0 +1,24 @@
# if building the kernel module in-tree, these config options
# should be put into a Kconfig file.
# if building out-of-tree, defining them here is as good as any.
# CONFIG_VHA_NEXEF_PLAT: build the VHA 3NX-F platform driver
export CONFIG_VHA_NEXEF_PLAT := m
ifeq ($(CONFIG_NEXEF_NNPU_INCLUDE),)
$(error CONFIG_NEXEF_NNPU_INCLUDE not set: You must provide the path to the NNPU tc_drv.h file in)
endif
ccflags-y += -I$(CONFIG_NEXEF_NNPU_INCLUDE)
ccflags-y += -I$(src)/../include
ccflags-y += -I$(src)/
ccflags-y += -Wall -g
# This should not be needed, but on some platform (especially old kernel)
# the module can't be build without it.
ccflags-y += -DRETPOLINE -Wfatal-errors
# NOT WORKING FOR NOW, DO NOT ENABLE
#ccflags-y += -DCONFIG_SET_FPGA_CLOCK=y
obj-$(CONFIG_VHA_NEXEF_PLAT) += nexef_plat.o

110
drivers/nna/nexef_platform/README.md Normal file
View File

@@ -0,0 +1,110 @@
3NX-F base platform driver
==========================
This document is here to explain how to build the linux drivers for the PCI FPGA based version of the 3NX-F.
It also expect you having basic knowledge on how to build the normal GPU driver and NNA driver.
### Building the drivers
##### 1. Getting the sources
First you need to have the NNA source tree (which you probably already have done as you read this file).
You will also need the NNPU (a GPU without graphic) driver that you can find here:
//powervr/swgraphics/rogueddk/MAIN/
##### 2. Building the base driver and NNA
Then you first build the NNA side driver which include the base 3NX-F driver: nexef_plat.ko
> To build the driver you will need first to locate the `tc_drv.h` file inside the GPU driver. It is normally located in
> `kernel/drivers/staging/imgtec/tc` and copy the full path of the folder containing it.
Start by building the NNA driver as usual:
- Creating a build folder
- Run cmake inside that build folder and configure as your needs
- run `make` inside the folder to build the userland part
Then we will build the three needed kernel modules:
- Go inside the `source/kernel/linux` folder
- run `make` with the proper arguments:
```
make -f Makefile.testing CONFIG_VHA_NEXEF=y CONFIG_HW_AX3=y CONFIG_NEXEF_NNPU_INCLUDE=/path/to/tc_drv/folder
```
It should results with three `.ko` files: `nexef_platform/nexef_plat.ko`, `img_mem/img_mem.ko` and `vha/vha.ko`.
Copy them to a handy location for later use.
##### 3. Building the NNPU driver
You now need to build the NNPU driver. This document is not an extensive explanation of how to build this driver.
Please refer to the GPU/NNPU documentation on the dependencies you need and how to build it.
So the main thing you will need is to set the proper environment variable to build the NNPU driver:
```
MIPS_ELF_ROOT=/path/to/mips/toolchain
RGX_BVNC=32.6.52.603
EXCLUDED_APIS=vulkan renderscript openrl opengles1 opengles3 camerahal composerhal memtrackhal sensorhal
PVR_BUILD_DIR=tc_linux
CLDNN=1
KERNELDIR=/path/to/linux/kernel/sources
ROGUEDDK_FOLDER=/path/to/nnpu/sources
NPU_BUILD_FOLDER=binary_${PVR_BUILD_DIR}_${BUILD}
NNPU_BUILD_FOLDER_PATH=/path/to/nnpu/sources/${NNPU_BUILD_FOLDER}
NNPU_FIRMWARE_FILE=${NNPU_BUILD_FOLDER}/target_neutral/rgx.fw.${RGX_BVNC}
NNPU_PVRSRVKM_MODULE=${NNPU_BUILD_FOLDER}/target_x86_64/pvrsrvkm.ko
SUPPORT_KMS=1
```
You would also probably need these variable to be set:
```
PVRVERSION_WITHOUT_P4=1
DRIVER_BRANCH=MAIN
PDUMP=0
SUPPORT_OPENCL_2_X=1
```
Then from the root NNPU driver you would just need to run `make -j8 imgdnn imgdnn_test build`
Once the NNPU driver is build you will need to install the required files, you can use the provided install.sh shell.
**Be careful** to not install the kernel modules and the rc script with it!
### Loading the drivers
To load the kernel drivers, you will need to load them in order.
But before loading the kernel modules, you need to set the DUT clock on the FPGA. For that you need the dbg_py tool
to be installed on the system you are using, and run the python script `set_fpga_freq.py` to set the FPGA clock.
Then you will be able to load the driver:
First the base 3NX-F driver:
`sudo insmod nexef_plat.ko`
Then load the NNA and NNPU driver:
```
sudo insmod img_mem.ko
sudo insmod vha.ko
sudo insmod pvrsrvkm.ko
```
**You must not** load the `tc.ko` as it will conflict with the base 3NX-F driver.
You also don't need the `drm_pdp.ko`file as there is no support for it in the 3NX-F base driver.
Info: The 3NX-F base driver basically replace the `tc.ko` module in this configuration.
##### What does NeXeF mean?
(*Or why does the base driver is called nexef?*)
*NeX* / *NeXeF* were coined by the GPT-2 Network (a transformer neural network) while trying to make it talking about
the 3NX-F. It added in the middle of a sentence "the 3NX (pronounced 3-Nex`)"
> **Imagination Technologies just released the 3NX-F**, a new "3NX" (pronounced "3-Nex") laser-based MEMS camera
> that has been designed to be "small, power-efficient, light-weight and with a compact size to be able to fit in a
> smartphone." The 3NX-F is an upgrade to the 1.5 million pixel 3NX that debuted in March.

1799
drivers/nna/nexef_platform/nexef_plat.c Normal file
View File

File diff suppressed because it is too large Load Diff

12
drivers/nna/nexef_platform/set_fpga_freq.py Normal file
View File

@@ -0,0 +1,12 @@
#!/usr/bin/env python
import sys
print sys.version
from dbg_py import *
if __name__ == "__main__":
config_devices()
set_dut_core_clk(25)
set_dut_iface_clk(25)

137
drivers/nna/vha/Makefile Normal file
View File

@@ -0,0 +1,137 @@
# Alias for backward compatibility
CONFIG_VHA_APOLLO := $(CONFIG_VHA_FPGA)
CONFIG_HW_AX3_MC := $(CONFIG_HW_MULTICORE)
obj-$(CONFIG_VHA) += vha.o
CONFIG_VHA_INFO := $(CONFIG_VHA)
obj-$(CONFIG_VHA_INFO) += vha_info.o
# Common files
vha-y := vha_api.o vha_common.o
vha-y += vha_dbg.o vha_pdump.o
ifeq ($(CONFIG_HW_AX3_MC), y)
subdir-ccflags-y += -I$(src)/multi
vha-y += multi/vha_dev.o multi/vha_wm.o multi/vha_mmu.o multi/vha_mt19937.o
else
subdir-ccflags-y += -I$(src)/single
vha-y += single/vha_dev.o single/vha_cnn.o single/vha_mmu.o
vha-$(CONFIG_HW_AX2) += single/vha_dev_ax2.o
vha-$(CONFIG_HW_AX3) += single/vha_dev_ax3.o
endif
PLAT := platform
vha-$(CONFIG_VHA_PCI) += $(PLAT)/vha_plat_pci.o
vha-$(CONFIG_VHA_DUMMY) += $(PLAT)/vha_plat_dummy.o
vha-$(CONFIG_VHA_DUMMY_HW_SIM) += $(PLAT)/vha_plat_dummy.o
vha-$(CONFIG_VHA_APOLLO) += $(PLAT)/vha_plat_apollo.o
vha-$(CONFIG_VHA_ODIN) += $(PLAT)/vha_plat_odin.o
vha-$(CONFIG_VHA_EMU) += $(PLAT)/vha_plat_emu.o
vha-$(CONFIG_VHA_FROST) += $(PLAT)/vha_plat_frost.o
vha-$(CONFIG_VHA_ORION) += $(PLAT)/vha_plat_orion.o
vha-$(CONFIG_VHA_NEXEF) += $(PLAT)/vha_plat_nexef.o
vha-$(CONFIG_VHA_DT_EXAMPLE) += $(PLAT)/vha_plat_dt_example.o $(PLAT)/vha_plat_dt.o
vha-$(CONFIG_VHA_THEAD_LIGHT_FPGA_C910) += $(PLAT)/vha_plat_thead.o $(PLAT)/vha_plat_thead_light_fpga_c910.o
vha-$(CONFIG_VHA_THEAD_LIGHT) += $(PLAT)/vha_plat_thead.o $(PLAT)/vha_plat_thead_light.o
# System configuration
# For AX2 set Mirage by default
CONFIG_VHA_SYS_MIRAGE := $(CONFIG_HW_AX2)
# For 3NX set to Aura by default.
CONFIG_VHA_SYS_AURA := $(CONFIG_HW_AX3)
# For 4NX(3NX-MC) set to Magna by default.
CONFIG_VHA_SYS_MAGNA := $(CONFIG_HW_AX3_MC)
ccflags-$(CONFIG_VHA_SYS_MIRAGE) += -DCFG_SYS_MIRAGE
ccflags-$(CONFIG_VHA_SYS_AURA) += -DCFG_SYS_AURA
ccflags-$(CONFIG_VHA_SYS_VAGUS) += -DCFG_SYS_VAGUS
ccflags-$(CONFIG_VHA_SYS_MAGNA) += -DCFG_SYS_MAGNA
ccflags-y += -I$(src)/$(PLAT)/
subdir-ccflags-y += -I$(src)
ifdef CONFIG_NEXEF_NNPU_INCLUDE
subdir-ccflags-y += -I${CONFIG_NEXEF_NNPU_INCLUDE}
endif
ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=VHA_CORE
ccflags-$(CONFIG_BUS_MASTERING) += -DFPGA_BUS_MASTERING
ccflags-$(CONFIG_VHA_DUMMY) += -DCONFIG_VHA_DUMMY
ccflags-$(CONFIG_VHA_DUMMY_HW_SIM) += -DCONFIG_VHA_DUMMY -DCONFIG_VHA_DUMMY_SIMULATE_HW_PROCESSING_TIME
ccflags-$(CONFIG_VHA_APOLLO) += -DCONFIG_VHA_APOLLO
ccflags-$(CONFIG_VHA_ORION) += -DCONFIG_VHA_ORION
ccflags-$(CONFIG_VHA_ODIN) += -DCONFIG_VHA_ODIN
ccflags-$(CONFIG_VHA_NEXEF) += -DCONFIG_VHA_NEXEF
ccflags-$(CONFIG_ION) += -Idrivers/staging/android/ion
ccflags-$(CONFIG_HW_AX2) += -DHW_AX2
# Mirrored page tables enabled by default if not specified
CONFIG_VHA_MMU_MIRRORED_CTX ?= y
ccflags-$(CONFIG_VHA_MMU_MIRRORED_CTX) += -DVHA_MMU_MIRRORED_CTX_SUPPORT
ccflags-$(CONFIG_HW_AX3) += -DHW_AX3
# Default cores
CONFIG_VHA_NCORES ?= 6
ccflags-$(CONFIG_HW_AX3_MC) += -DCONFIG_HW_MULTICORE -DCONFIG_VHA_NCORES=$(CONFIG_VHA_NCORES)
# Enhanced APM enabled by default if not specified
CONFIG_VHA_ENHANCED_APM ?= y
ccflags-$(CONFIG_VHA_ENHANCED_APM) += -DVHA_ENHANCED_APM
# Default OSID = 0
CONFIG_TARGET_OSID ?= 0
# Magna does not use multiple OSes approach
ifeq ($(CONFIG_HW_AX3), y)
ifeq ($(CONFIG_HW_AX3_MC),)
ccflags-y += -DOSID=$(CONFIG_TARGET_OSID)
endif
endif
# Enable Safety critical features by default
ifeq ($(CONFIG_HW_AX3_MC)$(CONFIG_VHA_SYS_VAGUS), y)
CONFIG_VHA_SCF ?= y
endif
ifeq ($(CONFIG_HW_AX3_MC)$(CONFIG_VHA_SCF), yy)
vha-y += multi/vha_sc_dbg.o
endif
ccflags-$(CONFIG_VHA_SCF) += -DVHA_SCF
# enable support for *_EVENT_INJECT registers only for MC
# and if kernel FUNCTION_ERROR_INJECTION is enabled
ifeq ($(CONFIG_FUNCTION_ERROR_INJECTION), y)
ifeq ($(CONFIG_HW_AX3), y)
ccflags-y += -DVHA_EVENT_INJECT
endif
endif
ifeq ($(CONFIG_FUNCTONAL_TEST_CONTROL), y)
ccflags-y += -DVHA_FUNCT_CTRL
endif
ifeq ($(CONFIG_FORCE_IOREG_DEBUG), y)
ccflags-y += -DVHA_FORCE_IO_DEBUG
endif
# Magna does not use multiple OSes approach
ifeq ($(CONFIG_HW_AX3), y)
ifeq ($(CONFIG_HW_AX3_MC),y)
CONFIG_VHA_LO_PRI_SUBSEGS = n
else
CONFIG_VHA_LO_PRI_SUBSEGS ?= n
endif
else
CONFIG_VHA_LO_PRI_SUBSEGS = n
endif
ccflags-$(CONFIG_VHA_LO_PRI_SUBSEGS) += -DVHA_USE_LO_PRI_SUB_SEGMENTS
ifeq ($(CONFIG_VHA_LO_PRI_SUBSEGS), y)
obj-$(CONFIG_VHA) += vha_monitor.o
endif

4213
drivers/nna/vha/multi/vha_dev.c Normal file
View File

File diff suppressed because it is too large Load Diff

261
drivers/nna/vha/multi/vha_mmu.c Normal file
View File

@@ -0,0 +1,261 @@
/*
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/moduleparam.h>
#include <linux/delay.h>
#include <uapi/vha.h>
#include "vha_common.h"
#include "vha_plat.h"
#include <vha_regs.h>
int vha_mmu_flush_ctx(struct vha_dev *vha, int ctx_id)
{
int ret;
uint64_t inval =
VHA_SET_FIELD_SIMPLE_VAL(OS0_MMU_CTRL_INVAL, PC, EN) |
VHA_SET_FIELD_SIMPLE_VAL(OS0_MMU_CTRL_INVAL, PD, EN) |
VHA_SET_FIELD_SIMPLE_VAL(OS0_MMU_CTRL_INVAL, PT, EN);
uint64_t pend = VHA_SET_FIELD_SIMPLE_VAL(OS0_MMU_CTRL_INVAL_STATUS, PENDING, EN);
/* No need to handle mmu cache, when core is already offline */
if (vha->state == VHA_STATE_OFF)
return 0;
#ifdef VHA_SCF
if (vha->hw_props.supported.parity && !vha->parity_disable) {
/* If pending bit is set then parity bit must be set as well ! */
pend |= VHA_SET_FIELD_SIMPLE_VAL(OS0_MMU_CTRL_INVAL_STATUS, PARITY, EN);
}
#endif
ret = IOPOLL64_CR_PDUMP_PARITY(0, 30, 150, pend, OS0_MMU_CTRL_INVAL_STATUS);
if (ret) {
dev_err(vha->dev, "Error during MMU ctx %d flush\n", ctx_id);
} else {
if (unlikely(ctx_id == VHA_INVALID_ID))
inval |= VHA_SET_FIELD_SIMPLE_VAL(OS0_MMU_CTRL_INVAL, ALL_CONTEXTS, EN);
else {
inval |= VHA_CR_SETBITS(OS0_MMU_CTRL_INVAL, CONTEXT, (uint64_t)ctx_id);
}
inval |= VHA_CR_SETBITS(OS0_MMU_CTRL_INVAL, CONTEXT, (uint64_t)ctx_id);
dev_dbg(vha->dev, "%s: ctx_id:%d (0x%llx)\n", __func__, ctx_id, inval);
img_pdump_printf("-- MMU invalidate TLB caches\n");
IOWRITE64_CR_PDUMP(inval, OS0_MMU_CTRL_INVAL);
}
return ret;
}
/* this function is called from img_mmu, to handle cache issues */
int vha_mmu_callback(enum img_mmu_callback_type callback_type,
int buf_id, void *data)
{
struct vha_session *session = data;
struct vha_dev *vha = session->vha;
int ctx_id;
int ret = 0;
if (!vha)
return 0;
for (ctx_id = 0; ctx_id < ARRAY_SIZE(session->mmu_ctxs); ctx_id++)
ret |= vha_mmu_flush_ctx(vha, session->mmu_ctxs[ctx_id].hw_id);
if (ret) {
dev_err(vha->dev, "Error during MMU flush (2), resetting the HW...\n");
/* Rollback commands being processed */
vha_rollback_cmds(vha);
/* Perform full reset */
vha_dev_stop(vha, true);
/* Reschedule */
vha_chk_cmd_queues(vha, true);
}
return ret;
}
static void do_mmu_ctx_setup(struct vha_dev *vha,
uint8_t hw_id, int pc_bufid, uint32_t pc_baddr)
{
img_pdump_printf("-- Setup MMU context:%d\n", hw_id);
IOWRITE64_CR_PDUMP(hw_id, OS0_MMU_CBASE_MAPPING_CONTEXT);
if (!vha->mmu_base_pf_test) {
IOWRITE64(vha->reg_base, VHA_CR_OS0_MMU_CBASE_MAPPING, pc_baddr);
/* This is physical address so we need use MEM_OS0:BLOCK tag
* when pdump'ing. */
img_pdump_printf("-- Setup MMU base address\n"
"WRW "_PMEM_":$0 "_PMEM_":BLOCK_%d:0 -- 'PC'\n"
"SHR "_PMEM_":$0 "_PMEM_":$0 %d\n"
"WRW64 :REG:%#x "_PMEM_":$0\n", pc_bufid,
IMG_MMU_PC_ADDR_SHIFT,
VHA_CR_OS0_MMU_CBASE_MAPPING);
dev_dbg(vha->dev, "%s: setting hardware ctx id:%u\n", __func__, hw_id);
} else
dev_info(vha->dev, "Bringup test: force MMU base page fault\n");
}
int vha_mmu_setup(struct vha_session *session)
{
struct vha_dev *vha = session->vha;
int ctx_id;
int ret = 0;
for (ctx_id = 0; ctx_id < ARRAY_SIZE(session->mmu_ctxs); ctx_id++)
dev_dbg(vha->dev,
"%s: mode:%d session ctxid:%x active ctxid:%x\n",
__func__, vha->mmu_mode,
session->mmu_ctxs[ctx_id].id,
vha->active_mmu_ctx);
if (vha->mmu_mode == VHA_MMU_DISABLED) {
img_pdump_printf("-- MMU bypass ON\n");
IOWRITE64_PDUMP(VHA_CR_OS(MMU_CTRL_BYPASS_EN),
VHA_CR_OS(MMU_CTRL));
return 0;
}
/* Using model context to track active context */
if (session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].id == vha->active_mmu_ctx)
return 0;
img_pdump_printf("-- MMU_SETUP_BEGIN\n");
img_pdump_printf("-- MMU bypass OFF\n");
IOWRITE64_PDUMP(0, VHA_CR_OS(MMU_CTRL));
for (ctx_id = 0; ctx_id < ARRAY_SIZE(session->mmu_ctxs); ctx_id++) {
do_mmu_ctx_setup(vha, session->mmu_ctxs[ctx_id].hw_id,
session->mmu_ctxs[ctx_id].pc_bufid,
session->mmu_ctxs[ctx_id].pc_baddr);
/* If there are multiple sessions using the same mmu hardware context
* we need to flush caches for the old context (id is the same).
* This will happen when number of processes is > VHA_MMU_MAX_HW_CTXS */
if (vha->mmu_ctxs[session->mmu_ctxs[ctx_id].hw_id] > 1) {
dev_dbg(vha->dev, "%s: flushing shared ctx id:%u\n",
__func__, session->mmu_ctxs[ctx_id].hw_id);
ret = vha_mmu_flush_ctx(vha, session->mmu_ctxs[ctx_id].hw_id);
if (ret) {
dev_err(vha->dev, "Error during MMU flush, resetting the HW...\n");
goto mmu_setup_err;
}
}
}
/* Using model context to track context change */
vha->active_mmu_ctx = session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].id;
dev_dbg(vha->dev, "%s: update ctx id active:%x pc:%#x\n",
__func__, vha->active_mmu_ctx,
session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].pc_baddr <<
VHA_CR_OS0_MMU_CBASE_MAPPING_BASE_ADDR_ALIGNSHIFT);
mmu_setup_err:
img_pdump_printf("-- MMU_SETUP_END\n");
return ret;
}
void vha_mmu_status(struct vha_dev *vha, uint8_t core_mask)
{
const char levels[][5] = {"PT", "PD", "PC", "BASE"};
uint32_t core_mmu_fault_reg_set_base = VHA_CR_CORE0_MMU_FAULT_STATUS1;
uint32_t core_mmu_fault_reg_set_offset =
VHA_CR_CORE1_MMU_FAULT_STATUS1 - VHA_CR_CORE0_MMU_FAULT_STATUS1;
uint32_t core_mmu_fault_status1_offset =
VHA_CR_CORE0_MMU_FAULT_STATUS1 - core_mmu_fault_reg_set_base;
uint32_t core_mmu_fault_status2_offset =
VHA_CR_CORE0_MMU_FAULT_STATUS2 - core_mmu_fault_reg_set_base;
while (core_mask) {
uint8_t id = ffs(core_mask) - 1;
uint64_t status1 = IOREAD64_REGIO(core_mmu_fault_reg_set_base +
id* core_mmu_fault_reg_set_offset +
core_mmu_fault_status1_offset);
uint64_t status2 = IOREAD64_REGIO(core_mmu_fault_reg_set_base +
id* core_mmu_fault_reg_set_offset +
core_mmu_fault_status2_offset);
#define MMU_FAULT_GETBITS(sreg, field, val) \
_get_bits(val, VHA_CR_CORE0_MMU_FAULT_ ## sreg ## _ ## field ## _SHIFT, \
~VHA_CR_CORE0_MMU_FAULT_ ## sreg ## _ ## field ## _CLRMSK)
uint64_t addr = MMU_FAULT_GETBITS(STATUS1, ADDRESS, status1);
uint8_t level = MMU_FAULT_GETBITS(STATUS1, LEVEL, status1);
uint8_t req_id = MMU_FAULT_GETBITS(STATUS1, REQ_ID, status1);
uint8_t ctx = MMU_FAULT_GETBITS(STATUS1, CONTEXT, status1);
uint8_t rnw = MMU_FAULT_GETBITS(STATUS1, RNW, status1);
uint8_t type = MMU_FAULT_GETBITS(STATUS1, TYPE, status1);
uint8_t fault = MMU_FAULT_GETBITS(STATUS1, FAULT, status1);
uint8_t bif_id = MMU_FAULT_GETBITS(STATUS2, BIF_ID, status2);
uint8_t tlb_entry = MMU_FAULT_GETBITS(STATUS2, TLB_ENTRY, status2);
uint8_t slc_bank = MMU_FAULT_GETBITS(STATUS2, BANK, status2);
uint64_t mapping = 0;
#undef MMU_FAULT_GETBITS
/* Select context and read current pc */
IOWRITE64_CR_REGIO(ctx, OS0_MMU_CBASE_MAPPING_CONTEXT);
mapping = IOREAD64_CR_REGIO(OS0_MMU_CBASE_MAPPING);
/* false alarm ? */
if (!fault)
return;
dev_dbg(vha->dev, "%s: Core%u MMU FAULT: s1:%llx s2:%llx\n",
__func__, id, status1, status2);
dev_warn(vha->dev, "%s: MMU fault while %s @ 0x%llx\n",
__func__, (rnw) ? "reading" : "writing", addr << 4);
dev_warn(vha->dev, "%s: level:%s Requestor:%x Context:%x Type:%s\n",
__func__, levels[level], req_id, ctx,
(type == 0) ? "VALID" :
(type == 2) ? "READ-ONLY" :
"UNKNOWN");
dev_warn(vha->dev, "%s: bif_id:%x tlb_entry:%x slc_bank:%x\n",
__func__, bif_id, tlb_entry, slc_bank);
dev_warn(vha->dev, "%s: current mapping@context%d:%#llx\n",
__func__, ctx,
mapping <<
VHA_CR_OS0_MMU_CBASE_MAPPING_BASE_ADDR_ALIGNSHIFT);
core_mask &= ~(VHA_CORE_ID_TO_MASK(id));
}
}

229
drivers/nna/vha/multi/vha_mt19937.c Normal file
View File

@@ -0,0 +1,229 @@
/*
A C-program for MT19937, with initialization improved 2002/1/26.
Coded by Takuji Nishimura and Makoto Matsumoto.
Copyright (C) 1997 - 2002, Makoto Matsumoto and Takuji Nishimura,
All rights reserved.
Copyright (C) 2005, Mutsuo Saito,
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. The names of its contributors may not be used to endorse or promote
products derived from this software without specific prior written
permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Any feedback is very welcome.
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html
email: m-mat @ math.sci.hiroshima-u.ac.jp (remove space)
Original code available at:
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html
*/
/*
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/kernel.h>
#include <linux/slab.h>
#include "vha_mt19937.h"
/* Period parameters */
#define N 624
#define M 397
#define MATRIX_A 0x9908b0dfUL /* constant vector a */
#define UPPER_MASK 0x80000000UL /* most significant w-r bits */
#define LOWER_MASK 0x7fffffffUL /* least significant r bits */
struct vha_mt19937_ctx
{
uint32_t mt[N]; /*!< The array for the state vector. */
int32_t mti; /*!< */
uint32_t mag01[2]; /*!< mag01[x] = x * MATRIX_A for x=0,1 */
};
int vha_mt19937_init(uint32_t seed, void **handle)
{
struct vha_mt19937_ctx *ctx;
/* Check input params. */
if (NULL == handle) {
pr_err("%s: invalid handle\n", __func__);
return -EINVAL;
}
/* Allocate SFMT context. */
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
if (ctx == NULL) {
pr_err("%s: failed to allocate context\n", __func__);
return -ENOMEM;
}
/* Initialise SFMT context with the seed. */
ctx->mt[0]= seed & 0xffffffffUL;
for (ctx->mti = 1; ctx->mti < N; ctx->mti++) {
ctx->mt[ctx->mti] =
(1812433253UL * (ctx->mt[ctx->mti - 1] ^ (ctx->mt[ctx->mti - 1] >> 30)) + ctx->mti);
/* See Knuth TAOCP Vol2. 3rd Ed. P.106 for multiplier. */
/* In the previous versions, MSBs of the seed affect */
/* only MSBs of the array mt[]. */
/* 2002/01/09 modified by Makoto Matsumoto */
ctx->mt[ctx->mti] &= 0xffffffffUL;
/* for >32 bit machines */
}
ctx->mag01[0] = 0x0UL;
ctx->mag01[1] = MATRIX_A;
*handle = ctx;
return 0;
}
int vha_mt19937_deinit(void *handle)
{
struct vha_mt19937_ctx *ctx = (struct vha_mt19937_ctx*)handle;
/* Check input params. */
if (NULL == handle) {
pr_err("%s: invalid handle\n", __func__);
return -EINVAL;
}
/* Free the SFMT context. */
kfree(ctx);
return 0;
}
int vha_mt19937_gen_uint32(void *handle, uint32_t *rand_val)
{
struct vha_mt19937_ctx *ctx = (struct vha_mt19937_ctx*)handle;
uint32_t y;
/* Check input params. */
if (NULL == handle) {
pr_err("%s: invalid handle\n", __func__);
return -EINVAL;
}
if (NULL == rand_val) {
pr_err("%s: invalid rand_val\n", __func__);
return -EINVAL;
}
/* Generate N words at one time. */
if (ctx->mti >= N) {
int kk;
for (kk = 0; kk < (N - M); kk++) {
y = (ctx->mt[kk] & UPPER_MASK) | (ctx->mt[kk + 1] & LOWER_MASK);
ctx->mt[kk] = ctx->mt[kk + M] ^ (y >> 1) ^ ctx->mag01[y & 0x1UL];
}
for (; kk < (N - 1); kk++) {
y = (ctx->mt[kk] & UPPER_MASK) | (ctx->mt[kk + 1] & LOWER_MASK);
ctx->mt[kk] = ctx->mt[kk + (M - N)] ^ (y >> 1) ^ ctx->mag01[y & 0x1UL];
}
y = (ctx->mt[N - 1] & UPPER_MASK) | (ctx->mt[0] & LOWER_MASK);
ctx->mt[N - 1] = ctx->mt[M - 1] ^ (y >> 1) ^ ctx->mag01[y & 0x1UL];
ctx->mti = 0;
}
y = ctx->mt[ctx->mti++];
/* Tempering */
y ^= (y >> 11);
y ^= (y << 7) & 0x9d2c5680UL;
y ^= (y << 15) & 0xefc60000UL;
y ^= (y >> 18);
*rand_val = y;
return 0;
}
int vha_mt19937_gen_range(void *handle, uint32_t min, uint32_t max, uint32_t *rand_val)
{
int ret;
uint32_t range_width;
/* Generate 32-bit random value. */
ret = vha_mt19937_gen_uint32(handle, rand_val);
if (ret != 0)
return ret;
/* Calculate the range. */
range_width = abs(max - min);
if (min > max)
{
min = max;
}
/* Calculate the random value within range. */
*rand_val = (range_width > 0) ? ((*rand_val % (range_width + 1)) + min) : min;
return 0;
}

93
drivers/nna/vha/multi/vha_mt19937.h Normal file
View File

@@ -0,0 +1,93 @@
/*
A C-program for MT19937, with initialization improved 2002/1/26.
Coded by Takuji Nishimura and Makoto Matsumoto.
Copyright (C) 1997 - 2002, Makoto Matsumoto and Takuji Nishimura,
All rights reserved.
Copyright (C) 2005, Mutsuo Saito
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. The names of its contributors may not be used to endorse or promote
products derived from this software without specific prior written
permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Any feedback is very welcome.
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html
email: m-mat @ math.sci.hiroshima-u.ac.jp (remove space)
Original code available at:
http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/MT2002/emt19937ar.html
*/
/*
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/types.h>
int vha_mt19937_init(uint32_t seed, void **handle);
int vha_mt19937_deinit(void *handle);
int vha_mt19937_gen_uint32(void *handle, uint32_t *rand_val);
int vha_mt19937_gen_range(void *handle, uint32_t min, uint32_t max, uint32_t *rand_val);

391
drivers/nna/vha/multi/vha_regs.h Normal file
View File

@@ -0,0 +1,391 @@
/*!
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef VHA_REGS_H
#define VHA_REGS_H
#include "../vha_io.h"
#if defined(HW_AX3)
#include <hwdefs/vha_cr_magna.h>
#else
#error "No HW layout defined"
#endif
#if defined(CFG_SYS_MAGNA)
#include <hwdefs/magna_system.h>
#endif
#define HW_SERIES (28U)
/* General register macros */
#define VHA_GET_FIELD_FULL_MASK(reg, field) \
(VHA_CR_BITMASK(reg, field) >> VHA_CR_##reg##_##field##_SHIFT)
#define VHA_SET_FIELD_SIMPLE_VAL(reg, field, type) \
(VHA_CR_##reg##_##field##_##type)
#define VHA_SET_FIELD_SIMPLE_FULL(reg, field) \
VHA_CR_BITMASK(reg, field)
#define VHA_WM_ID_TO_MASK(i) (1 << i)
#define VHA_CORE_ID_TO_MASK(i) (1 << i)
#define VHA_WM_MASK_TO_ID(m) ({ \
uint8_t _ret_ = m; \
WARN_ON(!_ret_); \
do { \
if (!_ret_) break; \
_ret_ = ffs(_ret_) - 1; \
} while (0); \
_ret_; \
})
#define VHA_CORE_MASK_TO_ID(m) VHA_WM_MASK_TO_ID(m)
#define VHA_CORE_MASK_TO_NUM(m) ({ \
uint8_t _ret_ = m; \
WARN_ON(!_ret_); \
do { \
if (!_ret_) break; \
_ret_ = _ret_ - ((_ret_ >> 1) & 0x55); \
_ret_ = (_ret_ & 0x33) + ((_ret_ >> 2) & 0x33); \
_ret_ = (((_ret_ + (_ret_ >> 4)) & 0x0F) * 0x01); \
} while (0); \
_ret_; \
})
#if defined(CONFIG_VHA_DUMMY)
#define VHA_LOCK_WM()
#define VHA_UNLOCK_WM()
#else
#define VHA_LOCK_WM() spin_lock_irq(&vha->irq_lock)
#define VHA_UNLOCK_WM() spin_unlock(&vha->irq_lock)
#endif
#define VHA_SELECT_WM(wm) ({ \
uint64_t reg = 0;\
uint8_t c = 10;\
IOWRITE64_CR_PDUMP(VHA_CR_SETBITS(TLC_WM_INDIRECT, ADDRESS, (uint64_t)wm), \
TLC_WM_INDIRECT); \
do { \
reg = IOREAD64_CR_REGIO(TLC_WM_INDIRECT); \
c--;\
} while ((reg != wm) && c > 0); \
WARN_ON(c == 0); \
})
/* Clock calibration defines. */
#define VHA_CALIBRATION_WM_ID 0
#define VHA_CALIBRATION_CORE_ID 0
#define VHA_CALIBRATION_CORE_MASK (1 << VHA_CALIBRATION_CORE_ID)
/* Event macro definitions */
#define VHA_SYS_EVENT_TYPE(name) \
VHA_CR_BITMASK(SYS_EVENT_TYPE, name)
#define VHA_WM_EVENT_TYPE(name) \
VHA_CR_BITMASK(WM_EVENT_TYPE, name)
#define VHA_CORE_EVENT_TYPE(name) \
VHA_CR_BITMASK(CORE_EVENT_TYPE, name)
#define VHA_IC_EVENT_TYPE(name) \
VHA_CR_BITMASK(INTERCONNECT_EVENT_TYPE, name)
#define VHA_SYS_EVENTS ( \
VHA_SYS_EVENT_TYPE(RAM_INIT_DONE ) | \
VHA_SYS_EVENT_TYPE(MEMBUS_RESET_DONE))
#ifdef VHA_SCF
#define VHA_SYS_SCF_ERR_EVENTS ( \
VHA_SYS_EVENT_TYPE(LOGIC_ERROR ) | \
VHA_SYS_EVENT_TYPE(RAM_CORRECTION ) | \
VHA_SYS_EVENT_TYPE(RAM_DETECTION ) | \
VHA_SYS_EVENT_TYPE(MMU_PARITY_ERROR) | \
VHA_SYS_EVENT_TYPE(AXI_MEMORY_PARITY_ERROR))
#else
#define VHA_SYS_SCF_ERR_EVENTS (0)
#endif
#define VHA_SYS_ERR_EVENTS ( \
VHA_SYS_SCF_ERR_EVENTS | \
VHA_SYS_EVENT_TYPE(LSYNC_INV_REQ ) | \
VHA_SYS_EVENT_TYPE(SYS_MEM_WDT ) | \
VHA_SYS_EVENT_TYPE(MMU_PAGE_FAULT ) | \
VHA_SYS_EVENT_TYPE(AXI_ERROR ))
#define VHA_SYS_EVENTS_DEFAULT ( \
VHA_SYS_EVENTS | \
VHA_SYS_ERR_EVENTS)
#define VHA_WM_EVENTS ( \
VHA_WM_EVENT_TYPE(RESPONSE_FIFO_READY))
#ifdef VHA_SCF
#define VHA_WM_SCF_ERR_EVENTS ( \
VHA_WM_EVENT_TYPE(LOGIC_FAULT ))
#else
#define VHA_WM_SCF_ERR_EVENTS (0)
#endif
#define VHA_WM_ERR_EVENTS ( \
VHA_WM_SCF_ERR_EVENTS | \
VHA_WM_EVENT_TYPE(WM_WL_WDT ) | \
VHA_WM_EVENT_TYPE(WM_WL_IDLE_WDT) | \
VHA_WM_EVENT_TYPE(WM_SOCIF_WDT ))
#define VHA_WM_EVENTS_DEFAULT ( \
VHA_WM_EVENTS | \
VHA_WM_ERR_EVENTS)
#define VHA_CORE_EVENTS ( \
VHA_CORE_EVENT_TYPE(CNN_COMPLETE ))
#ifdef VHA_SCF
#define VHA_CORE_SCF_ERR_EVENTS ( \
VHA_CORE_EVENT_TYPE(RAM_CORRECTION ) | \
VHA_CORE_EVENT_TYPE(RAM_DETECTION ) | \
VHA_CORE_EVENT_TYPE(LOGIC_ERROR ))
#else
#define VHA_CORE_SCF_ERR_EVENTS (0)
#endif
#define VHA_CORE_ERR_EVENTS ( \
VHA_CORE_SCF_ERR_EVENTS | \
VHA_CORE_EVENT_TYPE(CNN_ERROR ) | \
VHA_CORE_EVENT_TYPE(CORE_SYNC_ERROR ) | \
VHA_CORE_EVENT_TYPE(CORE_WDT ) | \
VHA_CORE_EVENT_TYPE(CORE_MEM_WDT ))
#define VHA_CORE_EVENTS_DEFAULT ( \
VHA_CORE_EVENTS | \
VHA_CORE_ERR_EVENTS)
#define VHA_IC_EVENTS (0)
#ifdef VHA_SCF
#define VHA_IC_SCF_ERR_EVENTS ( \
VHA_IC_EVENT_TYPE(LOGIC_ERROR ))
#else
#define VHA_IC_SCF_ERR_EVENTS (0)
#endif
#define VHA_IC_ERR_EVENTS ( \
VHA_IC_SCF_ERR_EVENTS | \
VHA_IC_EVENT_TYPE(LOCKSTEP_ERROR) | \
VHA_IC_EVENT_TYPE(SOCIF_READ_MISMATCH) | \
VHA_IC_EVENT_TYPE(SOCIF_READ_UNRESPONSIVE))
#define VHA_IC_EVENTS_DEFAULT ( \
VHA_IC_EVENTS | \
VHA_IC_ERR_EVENTS)
/* Clock macro definitions */
#define VHA_CLOCKS_MULTI_ALL 0xff
#define VHA_SPREAD_MASK(m) \
(((m * 0x0101010101010101ULL & 0x8040201008040201ULL) * \
0x0102040810204081ULL >> 49) & 0x5555)
#define VHA_SET_CLOCKS(mask, mode) \
(VHA_SPREAD_MASK(mask) * VHA_CR_SYS_CLK_CTRL0_MODE_##mode)
/* As REGBANK is a NOTOFF register, define its OFF to be AUTO by default. */
#define VHA_CR_SYS_CLK_CTRL0_REGBANK_OFF \
VHA_CR_SYS_CLK_CTRL0_REGBANK_AUTO
#define VHA_SYS_CLOCK_MODE(name, mode) \
VHA_CR_SYS_CLK_CTRL0_##name##_##mode
#define VHA_SYS_CLOCK_MODE_MULTI(name, mode, mask) \
(VHA_SET_CLOCKS(mask, mode) << VHA_CR_SYS_CLK_CTRL0_##name##0_SHIFT)
#define VHA_SYS_CLOCK_MODE_MULTI_ALL(name, mode) \
(VHA_SET_CLOCKS(VHA_CLOCKS_MULTI_ALL, mode) << \
VHA_CR_SYS_CLK_CTRL0_##name##0_SHIFT)
#define VHA_SYS_CLOCKS_DEFAULT(mode) ( (\
VHA_SYS_CLOCK_MODE(REGBANK, mode) | \
VHA_SYS_CLOCK_MODE(SOCM, mode) | \
VHA_SYS_CLOCK_MODE(LSYNC, mode) | \
VHA_SYS_CLOCK_MODE(SLC, mode) | \
VHA_SYS_CLOCK_MODE(AXI, mode) | \
VHA_SYS_CLOCK_MODE(INTERCONNECT, mode) | \
VHA_SYS_CLOCK_MODE_MULTI_ALL(WM, mode) | \
VHA_SYS_CLOCK_MODE_MULTI_ALL(NOC, mode) | \
VHA_SYS_CLOCK_MODE_MULTI_ALL(CORE, mode) \
) & VHA_CR_SYS_CLK_CTRL0_MASKFULL)
#define VHA_SYS_CLOCKS_RESET(mode) ( (\
VHA_SYS_CLOCK_MODE(REGBANK, mode) | \
VHA_SYS_CLOCK_MODE(SOCM, mode) | \
VHA_SYS_CLOCK_MODE(LSYNC, mode) | \
VHA_SYS_CLOCK_MODE(SLC, mode) | \
VHA_SYS_CLOCK_MODE(AXI, mode) | \
VHA_SYS_CLOCK_MODE(INTERCONNECT, mode) | \
VHA_SYS_CLOCK_MODE_MULTI_ALL(WM, mode) | \
VHA_SYS_CLOCK_MODE_MULTI_ALL(NOC, mode) \
) & VHA_CR_SYS_CLK_CTRL0_MASKFULL)
#define VHA_SYS_CLOCKS_CORE_FULL_MASK ( \
~(VHA_CR_SYS_CLK_CTRL0_CORE7_CLRMSK) | \
~(VHA_CR_SYS_CLK_CTRL0_CORE6_CLRMSK) | \
~(VHA_CR_SYS_CLK_CTRL0_CORE5_CLRMSK) | \
~(VHA_CR_SYS_CLK_CTRL0_CORE4_CLRMSK) | \
~(VHA_CR_SYS_CLK_CTRL0_CORE3_CLRMSK) | \
~(VHA_CR_SYS_CLK_CTRL0_CORE2_CLRMSK) | \
~(VHA_CR_SYS_CLK_CTRL0_CORE1_CLRMSK) | \
~(VHA_CR_SYS_CLK_CTRL0_CORE0_CLRMSK))
#define VHA_MAIN_CLOCK_MODE(name, mode) \
VHA_CR_CLK_CTRL0_##name##_##mode \
#define VHA_MAIN_CLOCKS_DEFAULT(mode) ( (\
VHA_MAIN_CLOCK_MODE(CNN_CORE_XBAR, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_MMM, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_EWO, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_PACK, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_OIN, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_POOL, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_SB, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_XBAR, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_NORM, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_ACT, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_ACCUM, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_CNV, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_CBUF, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_IBUF, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_CMD, mode) | \
VHA_MAIN_CLOCK_MODE(CNN, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_TRS_A, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_TRS_B, mode) | \
VHA_MAIN_CLOCK_MODE(MEMBUS_RESET, mode) | \
VHA_MAIN_CLOCK_MODE(BWM, mode) | \
VHA_MAIN_CLOCK_MODE(LOCM, mode) | \
VHA_MAIN_CLOCK_MODE(NOC, mode) | \
VHA_MAIN_CLOCK_MODE(ARB, mode) | \
VHA_MAIN_CLOCK_MODE(BIF, mode) \
) & VHA_CR_CLK_CTRL0_MASKFULL)
/* Response status macro definitions. */
#define VHA_WM_RESPONSE_STATUS(name) \
VHA_CR_BITMASK(WM_RESPONSE_FIFO_WL_STATUS, name)
#define VHA_WM_RESPONSE_SUCCESS ( \
VHA_WM_RESPONSE_STATUS(SUCCESS))
#define VHA_WM_RESPONSE_ERROR_CODE(name) \
(VHA_CR_WM_RESPONSE_FIFO_WL_STATUS_WL_ERROR_CODE_##name)
#define VHA_WM_RESPONSE_GET_ERROR_CODE(s) \
VHA_CR_GETBITS(WM_RESPONSE_FIFO_WL_STATUS, ERROR_CODE, s)
#define VHA_WM_RESPONSE_GET_FAILED_CORE_IDX(s) \
VHA_CR_GETBITS(WM_RESPONSE_FIFO_WL_STATUS, FAILED_CORE_IDX, s)
/* Core status macro definitions. */
#define VHA_CORE_STATUS(name) \
VHA_CR_BITMASK(CORE_EVENT_HOST_STATUS, name)
/* IC status macro definitions. */
#define VHA_IC_STATUS(name) \
VHA_CR_BITMASK(INTERCONNECT_EVENT_HOST_STATUS, name)
/* Confirmation writes error indication. */
/* There is a fake error bit set in the VHA_CR_WM_RESPONSE_FIFO_WL_STATUS reg
* to indicate confirmation writes error detected with software. */
#define VHA_REG_CONF_ERROR_SHIFT (63)
#define VHA_REG_CONF_ERROR_CLRMSK (0x7fffffffffffffffULL)
#define VHA_REG_CONF_ERROR_EN (0x8000000000000000ULL)
#define VHA_REG_SET_CONF_ERROR(r) \
(r |= VHA_REG_CONF_ERROR_EN)
#define VHA_REG_CLR_CONF_ERROR(r) \
(r &= VHA_REG_CONF_ERROR_CLRMSK)
#define VHA_REG_GET_CONF_ERROR(r) \
((r & VHA_REG_CONF_ERROR_EN) >> VHA_REG_CONF_ERROR_SHIFT)
/* Parity error indication. */
/* As not all regs with PARITY bit have LOGIC_ERROR bit to identify parity
* error, there's a fake parity bit set in these regs to indicate parity
* errors detected with software. */
#define VHA_REG_PARITY_ERROR_SHIFT (62)
#define VHA_REG_PARITY_ERROR_CLRMSK (0xBfffffffffffffffULL)
#define VHA_REG_PARITY_ERROR_EN (0x4000000000000000ULL)
#define VHA_REG_SET_PARITY_ERROR(r) \
(r |= VHA_REG_PARITY_ERROR_EN)
#define VHA_REG_CLR_PARITY_ERROR(r) \
(r &= VHA_REG_PARITY_ERROR_CLRMSK)
#define VHA_REG_GET_PARITY_ERROR(r) \
((r & VHA_REG_PARITY_ERROR_EN) >> VHA_REG_PARITY_ERROR_SHIFT)
/* Workload id mismatch indication. */
/* There is a fake error bit set in the VHA_CR_WM_RESPONSE_FIFO_WL_STATUS reg
* to indicate workload id mismatch error detected with software. */
#define VHA_REG_WL_ID_MISMATCH_ERROR_SHIFT (61)
#define VHA_REG_WL_ID_MISMATCH_ERROR_CLRMSK (0xDfffffffffffffffULL)
#define VHA_REG_WL_ID_MISMATCH_ERROR_EN (0x2000000000000000ULL)
#define VHA_REG_SET_WL_ID_MISMATCH_ERROR(r) \
(r |= VHA_REG_WL_ID_MISMATCH_ERROR_EN)
#define VHA_REG_CLR_WL_ID_MISMATCH_ERROR(r) \
(r &= VHA_REG_WL_ID_MISMATCH_ERROR_CLRMSK)
#define VHA_REG_GET_WL_ID_MISMATCH_ERROR(r) \
((r & VHA_REG_WL_ID_MISMATCH_ERROR_EN) >> VHA_REG_WL_ID_MISMATCH_ERROR_SHIFT)
/* CRC mismatch indication. */
/* There is a fake error bit set in the VHA_CR_WM_RESPONSE_FIFO_WL_STATUS reg
* to indicate workload id mismatch error detected with software. */
#define VHA_REG_COMBINED_CRC_ERROR_SHIFT (60)
#define VHA_REG_COMBINED_CRC_ERROR_CLRMSK (0xEFFFFFFFFFFFFFFFULL)
#define VHA_REG_COMBINED_CRC_ERROR_EN (0x1000000000000000ULL)
#define VHA_REG_SET_COMBINED_CRC_ERROR(r) \
(r |= VHA_REG_COMBINED_CRC_ERROR_EN)
#define VHA_REG_CLR_COMBINED_CRC_ERROR(r) \
(r &= VHA_REG_COMBINED_CRC_ERROR_CLRMSK)
#define VHA_REG_GET_COMBINED_CRC_ERROR(r) \
((r & VHA_REG_COMBINED_CRC_ERROR_EN) >> VHA_REG_COMBINED_CRC_ERROR_SHIFT)
/* General core error indication. */
/* There is a fake error bit set in the VHA_CR_WM_EVENT_STATUS reg
* to indicate that core error was detected for one of the assigned cores. */
#define VHA_REG_WM_CORE_ERROR_SHIFT (24)
#define VHA_REG_WM_CORE_ERROR_CLRMSK (0xfffffffffeffffffULL)
#define VHA_REG_WM_CORE_ERROR_EN (0x0000000001000000ULL)
#define VHA_REG_SET_WM_CORE_ERROR(r) \
(r |= VHA_REG_WM_CORE_ERROR_EN)
#define VHA_REG_CLR_WM_CORE_ERROR(r) \
(r &= VHA_REG_WM_CORE_ERROR_CLRMSK)
#define VHA_REG_GET_WM_CORE_ERROR(r) \
((r & VHA_REG_WM_CORE_ERROR_EN) >> VHA_REG_WM_CORE_ERROR_SHIFT)
/* General interconnect error indication. */
/* There is a fake error bit set in the VHA_CR_WM_EVENT_STATUS reg
* to indicate that interconnect error was detected for one of the assigned ones. */
#define VHA_REG_WM_IC_ERROR_SHIFT (25)
#define VHA_REG_WM_IC_ERROR_CLRMSK (0xfffffffffdffffffULL)
#define VHA_REG_WM_IC_ERROR_EN (0x0000000002000000ULL)
#define VHA_REG_SET_WM_IC_ERROR(r) \
(r |= VHA_REG_WM_IC_ERROR_EN)
#define VHA_REG_CLR_WM_IC_ERROR(r) \
(r &= VHA_REG_WM_IC_ERROR_CLRMSK)
#define VHA_REG_GET_WM_IC_ERROR(r) \
((r & VHA_REG_WM_IC_ERROR_EN) >> VHA_REG_WM_IC_ERROR_SHIFT)
#endif /* VHA_REGS_H */

264
drivers/nna/vha/multi/vha_sc_dbg.c Normal file
View File

@@ -0,0 +1,264 @@
/*
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/device.h>
#include <linux/gfp.h>
#include <linux/moduleparam.h>
#include <linux/list.h>
#include <linux/debugfs.h>
#include <linux/uaccess.h>
#include <uapi/vha.h>
#include "vha_common.h"
#include "vha_plat.h"
#include "vha_io.h"
#define VHA_MAX_NUM_SEGMENTS 10
static bool generate_crcs_enable = true;
module_param(generate_crcs_enable, bool, 0444);
MODULE_PARM_DESC(generate_crcs_enable,
"Enable generating safety CRCs");
struct vha_sc_dbgfs_ctx {
struct dentry *sc_debugfs_dir;
struct dentry *crcs_dir;
struct dentry *crcs_sub_dirs[VHA_MAX_NUM_SEGMENTS];
uint32_t num_cores_used[VHA_MAX_NUM_SEGMENTS];
uint32_t latest_crcs[VHA_MAX_NUM_SEGMENTS][VHA_NUM_CORES];
uint8_t segment_crc_idx_to_use;
uint8_t num_segments;
};
static ssize_t vha_bin_crcs_read(struct file *file, char __user *buf,
size_t count, loff_t *ppos)
{
struct vha_dev *vha = file->private_data;
uint32_t bin_crcs[VHA_MAX_NUM_SEGMENTS * VHA_MAX_CORES] = { 0 };
struct vha_sc_dbgfs_ctx *ctx = (struct vha_sc_dbgfs_ctx *)vha->sc_dbgfs_ctx;
size_t bytes = 0;
uint32_t offset = 0;
int i, j;
if (*ppos)
return 0;
for (i = 0; i < ctx->num_segments; i++) {
for (j = 0; j < ctx->num_cores_used[i]; j++) {
bin_crcs[offset] = ctx->latest_crcs[i][j];
offset++;
}
}
if (copy_to_user(buf, bin_crcs, offset * sizeof(bin_crcs[0]))) {
dev_err(vha->dev, "%s: bin_crcs read: copy to user failed\n",
__func__);
return -EFAULT;
}
if (count < offset)
return -EINVAL;
bytes = offset * sizeof(uint32_t);
*ppos = bytes;
return bytes;
}
static const struct file_operations vha_crcs_bin_fops = {
.owner = THIS_MODULE,
.open = simple_open,
.read = vha_bin_crcs_read,
};
static ssize_t vha_crcs_reset_write(struct file *file, const char __user *buf,
size_t count, loff_t *ppos)
{
struct vha_dev *vha = file->private_data;
struct vha_sc_dbgfs_ctx *ctx = (struct vha_sc_dbgfs_ctx *)vha->sc_dbgfs_ctx;
if (ctx->crcs_dir) {
int i = 0;
for (i = 0; i < VHA_MAX_NUM_SEGMENTS; i++)
if (ctx->crcs_sub_dirs[i]) {
debugfs_remove_recursive(ctx->crcs_sub_dirs[i]);
ctx->crcs_sub_dirs[i] = NULL;
}
ctx->segment_crc_idx_to_use = 0;
memset(ctx->latest_crcs, 0, sizeof(ctx->latest_crcs));
memset(ctx->num_cores_used, 0, sizeof(ctx->num_cores_used));
}
return count;
}
static const struct file_operations vha_crcs_reset_fops = {
.write = vha_crcs_reset_write,
.open = simple_open,
};
void vha_update_crcs(struct vha_dev *vha, uint32_t crcs[VHA_NUM_CORES], int n) {
struct vha_sc_dbgfs_ctx *ctx = (struct vha_sc_dbgfs_ctx *)vha->sc_dbgfs_ctx;
if (ctx->crcs_dir && ctx->num_segments) {
uint8_t i;
uint8_t crc_idx = ctx->segment_crc_idx_to_use;
char core_txt[7] = "core_x";
if (crc_idx >= VHA_MAX_NUM_SEGMENTS) {
dev_warn_once(vha->dev, "%s: unable to update crcs, too many segments\n", __func__);
return;
}
if (ctx->crcs_sub_dirs[crc_idx] == NULL) {
char dir_txt[10] = "segment_x";
snprintf(dir_txt, sizeof(dir_txt), "segment_%d", crc_idx);
ctx->crcs_sub_dirs[crc_idx] = debugfs_create_dir(dir_txt, ctx->crcs_dir);
for (i = 0; i < n; i++) {
snprintf(core_txt, sizeof(core_txt), "core_%d", i);
debugfs_create_x32(core_txt, S_IRUGO, ctx->crcs_sub_dirs[crc_idx],
&ctx->latest_crcs[crc_idx][i]);
}
}
ctx->num_cores_used[crc_idx] = n;
for (i = 0; i < n; i++)
ctx->latest_crcs[crc_idx][i] = crcs[i];
ctx->segment_crc_idx_to_use++;
if (ctx->segment_crc_idx_to_use >= ctx->num_segments)
ctx->segment_crc_idx_to_use = 0;
}
}
void vha_sc_dbg_init(struct vha_dev *vha, struct dentry *debugfs_dir)
{
struct vha_sc_dbgfs_ctx *ctx = devm_kzalloc(vha->dev,
sizeof(struct vha_sc_dbgfs_ctx), GFP_KERNEL);
if (!ctx) {
dev_err(vha->dev,
"%s: Out of memory when creating debugfs context!\n",
__func__);
return;
}
/* Create userspace node */
if (!debugfs_dir) {
dev_warn(vha->dev,
"%s: Probably debugfs not enabled in this kernel!\n",
__func__);
return;
}
#define VHA_DBGFS_CREATE_FILE_IN_DIR(_perm_, _name_, _fops_, dir) \
{ \
if (!debugfs_create_file(_name_, \
_perm_, ctx->dir, vha, \
&vha_##_fops_##_fops)) { \
dev_warn(vha->dev, \
"%s: failed to create %s dbg file!\n", \
__func__, _name_); \
} \
}
#if LINUX_VERSION_CODE >= KERNEL_VERSION(5,7,0)
#define CTX_DBGFS_CREATE_RW(_type_, _name_, _ctx_dev_member_, dir) \
{ \
struct dentry *dentry; \
debugfs_create_##_type_(_name_, \
S_IWUSR|S_IRUGO, ctx->dir, \
&ctx->_ctx_dev_member_); \
dentry = debugfs_lookup(_name_, ctx->dir); \
if (!dentry) { \
dev_warn(vha->dev, \
"%s: failed to create %s dbg file!\n", \
__func__, _name_); \
} else { \
dput(dentry); \
} \
}
#else
#define CTX_DBGFS_CREATE_RW(_type_, _name_, _ctx_dev_member_, dir) \
{ \
if (!debugfs_create_##_type_(_name_, \
S_IWUSR|S_IRUGO, ctx->dir, \
&ctx->_ctx_dev_member_)) { \
dev_warn(vha->dev, \
"%s: failed to create %s dbg file!\n", \
__func__, _name_); \
} \
}
#endif
ctx->sc_debugfs_dir = debugfs_create_dir("sf_gen", debugfs_dir);
CTX_DBGFS_CREATE_RW(u8, "num_segments", num_segments, sc_debugfs_dir);
if (generate_crcs_enable) {
ctx->crcs_dir = debugfs_create_dir("CRCs", ctx->sc_debugfs_dir);
if (ctx->crcs_dir) {
VHA_DBGFS_CREATE_FILE_IN_DIR(S_IWUSR, "crcs_reset", crcs_reset, crcs_dir);
VHA_DBGFS_CREATE_FILE_IN_DIR(S_IRUGO, "crcs_bin", crcs_bin, crcs_dir);
}
}
#undef VHA_DBGFS_CREATE_FILE_IN_DIR
#undef CTX_DBGFS_CREATE_RW
vha->sc_dbgfs_ctx = (void *)ctx;
}
void vha_sc_dbg_deinit(struct vha_dev *vha)
{
struct vha_sc_dbgfs_ctx *ctx =
(struct vha_sc_dbgfs_ctx *)vha->sc_dbgfs_ctx;
debugfs_remove_recursive(ctx->sc_debugfs_dir);
}

1896
drivers/nna/vha/multi/vha_wm.c Normal file
View File

File diff suppressed because it is too large Load Diff

104
drivers/nna/vha/platform/vha_plat.h Normal file
View File

@@ -0,0 +1,104 @@
/*!
*****************************************************************************
*
* @File vha_plat.h
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef VHA_PLAT_H
#define VHA_PLAT_H
/* Core clock frequency: default 30MHz */
#define VHA_CORE_CLOCK_MHZ 30
/* Core watchdog cycles default value */
#if defined(HW_AX2)
#define VHA_CORE_WDT_CYCLES 0x7fffff
#elif defined(HW_AX3)
/* MMM can transfer any number of bytes at cost of higher cycles, setting it to ~100ms @800MHz */
#define VHA_CORE_WDT_CYCLES 0xfffffff
/* Memory watchdog is set ~1ms @800MHz which is very safe value to avoid any false interrupts */
#define VHA_CORE_MEM_WDT_CYCLES 0xffffffff
#endif
#ifdef CONFIG_HW_MULTICORE
/* System watchdog cycles default values */
#define VHA_SYS_MEM_WDT_CYCLES 0xffffffff
/* WM watchdog cycles default values */
#define VHA_WM_WDT_CYCLES 0xffffffff
#define VHA_WM_IDLE_WDT_CYCLES 0xfffff
#define VHA_WM_SOCIF_WDT_CYCLES 0xfffff
/* Core watchdog cycles default values */
/* MMM can transfer any number of bytes at cost of higher cycles, setting it to ~100ms @800MHz */
#define VHA_CORE_WDT_CYCLES 0xfffffff
/* Memory watchdog is set ~1ms @800MHz which is very safe value to avoid any false interrupts */
#define VHA_CORE_MEM_WDT_CYCLES 0xffffffff
#define VHA_CORE_SYNC_WDT_CYCLES 0xffff
#endif
/* Memory burst size */
#define VHA_CORE_MH_MAX_BURST_LENGTH 128
/* SLC cache policy type (0-use cache, 1-bypass cache) */
#define VHA_CORE_MH_SLC_CACHE_POLICY_TYPE 1
/* GPU pipe coherent type */
#define VHA_CORE_MH_GPU_PIPE_COHERENT_TYPE 1
/* Persistence priority 0-lowest,3-highest */
#define VHA_CORE_MH_PERSISTENCE_PRIO 0
/* Suspend delay in ms after which the
* runtime suspend callback is called */
#define VHA_CORE_SUSPEND_DELAY 10
/* Default OCM start address */
#ifdef CONFIG_HW_MULTICORE
#define VHA_OCM_ADDR_START 0x1000
#else
#define VHA_OCM_ADDR_START (~0)
#endif
/* IO hooks */
uint64_t vha_plat_read64(void *addr);
void vha_plat_write64(void *addr, uint64_t val);
int vha_plat_init(void);
int vha_plat_deinit(void);
#endif /* VHA_PLAT_H */

862
drivers/nna/vha/platform/vha_plat_apollo.c Normal file
View File

@@ -0,0 +1,862 @@
/*!
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/module.h>
#include <linux/device.h>
#include <linux/gfp.h>
#include <linux/version.h>
#if LINUX_VERSION_CODE < KERNEL_VERSION(5,10,0)
#include <linux/dma-mapping.h>
#else
#include <linux/dma-map-ops.h>
#endif
#include <linux/pci.h>
#include <linux/pm.h>
#include <linux/pm_runtime.h>
#include <linux/mod_devicetable.h>
#include "uapi/version.h"
#include "vha_common.h"
#include "vha_plat.h"
#define DEVICE_NAME "vha"
#define IS_APOLLO_DEVICE(devid) ((devid) == PCI_APOLLO_DEVICE_ID)
/*
* from TCF Support FPGA.Technical Reference
* Manual.1.0.92.Internal Atlas GEN.External.doc:
*/
/* Altas - System control register bar */
#define PCI_ATLAS_SYS_CTRL_REGS_BAR (0)
/* Altas - System control register offset */
#define PCI_ATLAS_SYS_CTRL_REGS_OFFSET (0x0000)
/* Atlas - Offset of INTERRUPT_STATUS */
/*#define PCI_ATLAS_INTERRUPT_STATUS (0x00E0)*/
/* Atlas - Offset of INTERRUPT_ENABLE */
/*#define PCI_ATLAS_INTERRUPT_ENABLE (0x00F0)*/
/* Atlas - Offset of INTERRUPT_CLEAR */
/*#define PCI_ATLAS_INTERRUPT_CLEAR (0x00F8)*/
/* Atlas - Master interrupt enable */
#define PCI_ATLAS_MASTER_ENABLE (1<<31)
/* Atlas - Device interrupt */
#define PCI_ATLAS_DEVICE_INT (1<<13)
/* Atlas - SCB Logic soft reset */
#define PCI_ATLAS_SCB_RESET (1<<4)
/* Atlas - PDP2 soft reset */
#define PCI_ATLAS_PDP2_RESET (1<<3)
/* Atlas - PDP1 soft reset */
#define PCI_ATLAS_PDP1_RESET (1<<2)
/* Atlas - soft reset the DDR logic */
#define PCI_ATLAS_DDR_RESET (1<<1)
/* Atlas - soft reset the device under test */
#define PCI_ATLAS_DUT_RESET (1<<0)
#define PCI_ATLAS_RESET_REG_OFFSET (0x0080)
#define PCI_ATLAS_RESET_BITS (PCI_ATLAS_DDR_RESET | PCI_ATLAS_DUT_RESET \
| PCI_ATLAS_PDP1_RESET | PCI_ATLAS_PDP2_RESET | \
PCI_ATLAS_SCB_RESET)
/* Apollo - Offset of INTERRUPT_STATUS */
#define PCI_APOLLO_INTERRUPT_STATUS (0x00C8)
/* Apollo - Offset of INTERRUPT_ENABLE */
#define PCI_APOLLO_INTERRUPT_ENABLE (0x00D8)
/* Apollo - Offset of INTERRUPT_CLEAR */
#define PCI_APOLLO_INTERRUPT_CLEAR (0x00E0)
/* Apollo - DCM Logic soft reset */
#define PCI_APOLLO_DCM_RESET (1<<10)
#define PCI_APOLLO_RESET_BITS (PCI_ATLAS_RESET_BITS | PCI_APOLLO_DCM_RESET)
#define PCI_ATLAS_TEST_CTRL (0xb0)
#define PCI_APOLLO_TEST_CTRL (0x98)
#define PCI_ATLAS_VENDOR_ID (0x1010)
#define PCI_ATLAS_DEVICE_ID (0x1CF1)
#define PCI_APOLLO_DEVICE_ID (0x1CF2)
/* Number of core cycles used to measure the core clock frequency */
#define FREQ_MEASURE_CYCLES 0x7fffff
/*#define FPGA_IMAGE_REV_OFFSET (0x604)
#define FPGA_IMAGE_REV_MASK (0xFFFF)*/
/* Parameters applicable when using bus master mode */
static unsigned long contig_phys_start;
module_param(contig_phys_start, ulong, 0444);
MODULE_PARM_DESC(contig_phys_start,
"Physical address of start of contiguous region");
static uint32_t contig_size;
module_param(contig_size, uint, 0444);
MODULE_PARM_DESC(contig_size,
"Size of contiguous region: takes precedence over any PCI based memory");
static uint32_t fpga_heap_type = IMG_MEM_HEAP_TYPE_UNIFIED;
module_param(fpga_heap_type, uint, 0444);
MODULE_PARM_DESC(fpga_heap_type, "Fpga primary heap type");
static unsigned long pci_size;
module_param(pci_size, ulong, 0444);
MODULE_PARM_DESC(pci_size,
"physical size in bytes, when 0 (default), use all memory in the PCI bar");
static unsigned long pci_offset;
module_param(pci_offset, ulong, 0444);
MODULE_PARM_DESC(pci_offset, "offset from PCI bar start. (default: 0)");
static bool mem_static_kptr = true;
module_param(mem_static_kptr, bool, 0444);
MODULE_PARM_DESC(mem_static_kptr,
"Creates static kernel mapping for fpga memory");
/*
* Special handling (not implemented) is required for the VHA device
* to be able to access both carvout buffers (internal memory) and
* dmabuf buffers (system memory).The latter have to go through
* the system bus to be accessed whereas the former do not.
*/
static struct heap_config vha_plat_fpga_heap_configs[] = {
/* Primary heap used for internal allocations */
#ifdef FPGA_BUS_MASTERING
{
.type = -1, /* selected with fpga_heap_type */
.options = {
.unified.gfp_type = GFP_DMA32 | __GFP_ZERO,
.unified.max_order = 4,
},
.to_dev_addr = NULL,
.to_host_addr = NULL,
},
#elif CONFIG_GENERIC_ALLOCATOR
{
.type = IMG_MEM_HEAP_TYPE_CARVEOUT,
/* .options.carveout to be filled at run time */
/* .to_dev_addr to be filled at run time */
/* .to_host_addr to be filled at run time */
},
#else
#error Neither FPGA_BUS_MASTERING or CONFIG_GENERIC_ALLOCATOR was defined
#endif
/* Secondary heap used for importing an external memory */
#ifdef FPGA_BUS_MASTERING
{
.type = IMG_MEM_HEAP_TYPE_ANONYMOUS,
},
#endif
#if CONFIG_DMA_SHARED_BUFFER
{
.type = IMG_MEM_HEAP_TYPE_DMABUF,
.to_dev_addr = NULL,
#ifndef FPGA_BUS_MASTERING
.options.dmabuf = {
.use_sg_dma = true,
},
#endif
},
#endif
};
static const int vha_plat_fpga_heaps =
sizeof(vha_plat_fpga_heap_configs)/sizeof(*vha_plat_fpga_heap_configs);
static const struct pci_device_id pci_pci_ids[] = {
{ PCI_DEVICE(PCI_ATLAS_VENDOR_ID, PCI_ATLAS_DEVICE_ID), },
{ PCI_DEVICE(PCI_ATLAS_VENDOR_ID, PCI_APOLLO_DEVICE_ID), },
{ 0, }
};
MODULE_DEVICE_TABLE(pci, pci_pci_ids);
struct imgpci_prvdata {
int irq;
struct {
unsigned long addr;
unsigned long size;
void __iomem *km_addr;
} memmap[3];
struct pci_dev *pci_dev;
};
struct img_pci_driver {
struct pci_dev *pci_dev;
struct pci_driver pci_driver;
};
static int vha_plat_probe(struct pci_dev *pci_dev,
const struct pci_device_id *id);
static void vha_plat_remove(struct pci_dev *dev);
static int vha_plat_suspend(struct device *dev);
static int vha_plat_resume(struct device *dev);
static int vha_plat_runtime_idle(struct device *dev);
static int vha_plat_runtime_suspend(struct device *dev);
static int vha_plat_runtime_resume(struct device *dev);
static struct dev_pm_ops vha_pm_plat_ops = {
#ifdef FPGA_BUS_MASTERING
/* Runtime pm will not work with fpga internal memory
* because pci bus driver suspend is also called,
* which disables core/mem clocks */
SET_RUNTIME_PM_OPS(vha_plat_runtime_suspend,
vha_plat_runtime_resume, vha_plat_runtime_idle)
#endif
SET_SYSTEM_SLEEP_PM_OPS(vha_plat_suspend, vha_plat_resume)
};
static ssize_t info_show(struct device_driver *drv, char *buf)
{
return sprintf(buf, "VHA FPGA driver version : " VERSION_STRING "\n");
}
static DRIVER_ATTR_RO(info);
static struct attribute *drv_attrs[] = {
&driver_attr_info.attr,
NULL
};
ATTRIBUTE_GROUPS(drv);
static struct img_pci_driver vha_pci_drv = {
.pci_driver = {
.name = "vha_pci",
.id_table = pci_pci_ids,
.probe = vha_plat_probe,
.remove = vha_plat_remove,
.driver = {
.pm = &vha_pm_plat_ops,
.groups = drv_groups,
}
},
};
static ulong maxmapsizeMB = (sizeof(void *) == 4) ? 400 : 1024;
static int interrupt_status_reg = -1;
static int interrupt_clear_reg = -1;
static int interrupt_enable_reg = -1;
static int test_ctrl_reg = -1;
static unsigned int fpga_readreg32(struct imgpci_prvdata *data,
int bar, unsigned long offset
)
{
void __iomem *reg =
(void __iomem *)(data->memmap[bar].km_addr + offset);
return ioread32(reg);
}
static void fpga_writereg32(struct imgpci_prvdata *data,
int bar, unsigned long offset, int val)
{
void __iomem *reg =
(void __iomem *)(data->memmap[bar].km_addr + offset);
iowrite32(val, reg);
}
static void reset_fpga(struct pci_dev *dev,
struct imgpci_prvdata *data, unsigned int mask)
{
uint32_t bits = 0;
if (!dev)
return;
bits = PCI_APOLLO_RESET_BITS;
dev_dbg(&dev->dev, "reset fpga!\n");
bits &= mask;
if (bits) {
uint32_t val = fpga_readreg32(data, 0, PCI_ATLAS_RESET_REG_OFFSET);
val &= ~bits;
fpga_writereg32(data, 0, PCI_ATLAS_RESET_REG_OFFSET, val);
udelay(100); /* arbitrary delays, just in case! */
val |= bits;
fpga_writereg32(data, 0, PCI_ATLAS_RESET_REG_OFFSET, val);
/* If not only DUT is reset, add a delay */
if (mask != PCI_ATLAS_DUT_RESET)
msleep(100);
else
udelay(100); /* arbitrary delays, just in case! */
}
dev_dbg(&dev->dev, "reset fpga done!\n");
}
static void fpga_clear_irq(struct imgpci_prvdata *data, unsigned int intstatus)
{
unsigned int max_retries = 1000;
while (fpga_readreg32(data, PCI_ATLAS_SYS_CTRL_REGS_BAR,
interrupt_status_reg) && max_retries--)
fpga_writereg32(data, PCI_ATLAS_SYS_CTRL_REGS_BAR,
interrupt_clear_reg,
(PCI_ATLAS_MASTER_ENABLE | intstatus));
}
static irqreturn_t pci_thread_irq(int irq, void *dev_id)
{
struct pci_dev *dev = (struct pci_dev *)dev_id;
return vha_handle_thread_irq(&dev->dev);
}
static irqreturn_t pci_isrcb(int irq, void *dev_id)
{
unsigned int intstatus;
struct pci_dev *dev = (struct pci_dev *)dev_id;
struct imgpci_prvdata *data = vha_get_plat_data(&dev->dev);
irqreturn_t ret = IRQ_NONE;
if (data == NULL || dev_id == NULL) {
/* spurious interrupt: not yet initialised. */
goto exit;
}
intstatus = fpga_readreg32(data,
PCI_ATLAS_SYS_CTRL_REGS_BAR,
interrupt_status_reg);
if (intstatus) {
ret = vha_handle_irq(&dev->dev);
/*
* We need to clear interrupts for the embedded device
* via the fpga interrupt controller...
*/
fpga_clear_irq(data, intstatus);
} else {
/* either a spurious interrupt, or, more likely
* a shared interrupt line, which will be handled by another driver
*/
goto exit;
}
exit:
return ret;
}
/*
* IO hooks : Address bus for hw registers is 32bit!
*/
uint64_t vha_plat_read64(void *addr)
{
return (uint64_t)readl((const volatile void __iomem *)addr) |
((uint64_t)readl((const volatile void __iomem *)addr + 4) << 32);
}
void vha_plat_write64(void *addr, uint64_t val)
{
writel((uint32_t)(val & 0xffffffff), (volatile void __iomem *)addr);
writel((uint32_t)(val >> 32), (volatile void __iomem *)addr + 4);
}
int vha_plat_deinit(void)
{
struct pci_dev *dev = vha_pci_drv.pci_dev;
int ret;
if (dev) {
struct imgpci_prvdata *data = vha_get_plat_data(&dev->dev);
if (data) {
/* reset the hardware */
reset_fpga(data->pci_dev, data, ~0);
} else {
dev_dbg(&dev->dev,
"%s: prv data not found, HW reset omitted\n",
__func__);
}
} else {
pr_debug("%s: dev missing, HW reset omitted\n", __func__);
}
/* Unregister the driver from the OS */
pci_unregister_driver(&(vha_pci_drv.pci_driver));
ret = vha_deinit();
if (ret)
pr_err("VHA driver deinit failed\n");
return ret;
}
#ifdef CONFIG_GENERIC_ALLOCATOR
static phys_addr_t carveout_to_dev_addr(union heap_options *options,
phys_addr_t addr)
{
phys_addr_t base = options->carveout.phys;
size_t size = options->carveout.size;
unsigned long offset = options->carveout.offs;
if (addr - offset >= base && addr < base + size - offset)
return addr - base;
pr_err("%s: unexpected addr! base 0x%llx size %zu offs %zu addr 0x%llx\n",
__func__, base, size, offset, addr);
WARN_ON(1);
return addr;
}
static phys_addr_t carveout_to_host_addr(union heap_options *options,
phys_addr_t addr)
{
phys_addr_t base = options->carveout.phys;
size_t size = options->carveout.size;
unsigned long offset = options->carveout.offs;
if (addr < size - offset)
return base + addr;
pr_err("%s: unexpected addr! base %llx size %zu offs %zu addr %#llx\n",
__func__, base, size, offset, addr);
WARN_ON(1);
return addr;
}
static void *carveout_get_kptr(phys_addr_t addr,
size_t size, enum img_mem_attr mattr)
{
/*
* Device memory is I/O memory and as a rule, it cannot
* be dereferenced safely without memory barriers, that
* is why it is guarded by __iomem (return of ioremap)
* and checked by sparse. It is accessed only through
* ioread32(), iowrit32(), etc.
*
* In x86 this memory can be dereferenced and safely
* accessed, i.e. a * __iomem pointer can be casted to
* a regular void* * pointer. We cast this here
* assuming FPGA is x86 and add __force to silence the
* sparse warning
*
* Note: System memory carveout can be used with cached turned on.
* */
void *kptr = NULL;
if (mattr & IMG_MEM_ATTR_UNCACHED)
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
kptr = (void * __force *)ioremap_nocache(addr, size);
#else
kptr = (void * __force *)ioremap(addr, size);
#endif
else if (mattr & IMG_MEM_ATTR_CACHED)
kptr = (void * __force *)ioremap_cache(addr, size);
else if (mattr & IMG_MEM_ATTR_WRITECOMBINE)
kptr = (void * __force *)ioremap_wc(addr, size);
return kptr;
}
static int carveout_put_kptr(void *addr)
{
iounmap((volatile void __iomem *)addr);
return 0;
}
#endif
static int vha_plat_probe(struct pci_dev *pci_dev,
const struct pci_device_id *id)
{
int bar, ret = 0;
struct imgpci_prvdata *data;
size_t maxmapsize = maxmapsizeMB * 1024 * 1024;
struct device *dev = &pci_dev->dev;
int heap;
dev_dbg(dev, "probing device, pci_dev\n");
/* Enable the device */
if (pci_enable_device(pci_dev))
goto out_free;
if (dev->dma_mask) {
dev_info(dev, "%s dev->dma_mask : %#llx\n",
__func__, *dev->dma_mask);
} else {
dev_info(dev, "%s mask unset, setting coherent\n", __func__);
dev->dma_mask = &dev->coherent_dma_mask;
}
dev_info(dev, "%s dma_set_mask %#llx\n", __func__, dma_get_mask(dev));
ret = dma_set_mask(dev, dma_get_mask(dev));
if (ret) {
dev_err(dev, "%s failed to set dma mask\n", __func__);
goto out_disable;
}
/* Reserve PCI I/O and memory resources */
if (pci_request_regions(pci_dev, "imgpci"))
goto out_disable;
/* Create a kernel space mapping for each of the bars */
data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
dev_dbg(dev, "allocated imgpci_prvdata @ %p\n", data);
memset(data, 0, sizeof(*data));
for (bar = 0; bar < 3; bar++) {
data->memmap[bar].size = pci_resource_len(pci_dev, bar);
data->memmap[bar].addr = pci_resource_start(pci_dev, bar);
if (bar == 2) {
if (pci_size)
data->memmap[bar].size = pci_size;
/* ioremap fpga memory only when static mode is used */
if (!mem_static_kptr)
continue;
}
if (data->memmap[bar].size > maxmapsize) {
/*
* We avoid mapping too big regions: we do not need
* such a big amount of memory and some times we do
* not have enough contiguous 'vmallocable' memory.
*/
dev_warn(dev, "not mapping all mem for bar %u\n", bar);
data->memmap[bar].size = maxmapsize;
}
data->memmap[bar].km_addr = devm_ioremap(dev,
pci_resource_start(pci_dev, bar),
data->memmap[bar].size);
dev_dbg(dev, "[bar %u] addr: 0x%lx size: 0x%lx km: 0x%p\n",
bar, data->memmap[bar].addr,
data->memmap[bar].size,
data->memmap[bar].km_addr);
}
/* Get the IRQ...*/
data->irq = pci_dev->irq;
data->pci_dev = pci_dev;
vha_pci_drv.pci_dev = pci_dev;
reset_fpga(pci_dev, data, ~0);
interrupt_status_reg = PCI_ATLAS_SYS_CTRL_REGS_OFFSET +
PCI_APOLLO_INTERRUPT_STATUS;
interrupt_clear_reg = PCI_ATLAS_SYS_CTRL_REGS_OFFSET +
PCI_APOLLO_INTERRUPT_CLEAR;
interrupt_enable_reg = PCI_ATLAS_SYS_CTRL_REGS_OFFSET +
PCI_APOLLO_INTERRUPT_ENABLE;
test_ctrl_reg = PCI_ATLAS_SYS_CTRL_REGS_OFFSET +
PCI_APOLLO_TEST_CTRL;
/*
* We need to enable interrupts for the embedded device
* via the fpga interrupt controller...
*/
{
unsigned int ena;
ena = fpga_readreg32(data, PCI_ATLAS_SYS_CTRL_REGS_BAR,
interrupt_enable_reg);
ena |= PCI_ATLAS_MASTER_ENABLE | PCI_ATLAS_DEVICE_INT;
fpga_writereg32(data, PCI_ATLAS_SYS_CTRL_REGS_BAR,
interrupt_enable_reg, ena);
fpga_clear_irq(data, ena);
}
#ifdef FPGA_BUS_MASTERING
dev_dbg(dev, "enabling FPGA bus mastering\n");
fpga_writereg32(data, PCI_ATLAS_SYS_CTRL_REGS_BAR, test_ctrl_reg, 0x0);
#else
/* Route to internal RAM - this is reset value */
dev_dbg(dev, "disabling FPGA bus mastering\n");
fpga_writereg32(data, PCI_ATLAS_SYS_CTRL_REGS_BAR, test_ctrl_reg, 0x1);
#endif
/* patch heap config with PCI memory addresses */
for (heap = 0; heap < vha_plat_fpga_heaps; heap++) {
struct heap_config *cfg = &vha_plat_fpga_heap_configs[heap];
#ifdef CONFIG_GENERIC_ALLOCATOR
if (cfg->type == IMG_MEM_HEAP_TYPE_CARVEOUT) {
if (contig_size && contig_phys_start) {
/*
* 2 types of carveout memory are supported:
* memory carved out of the main DDR
* memory region.
* eg: linux boot option memmap=512M$0x5CAFFFFF
* This is configured using module parameters:
* contig_phys_start and size
* DDR populated in the actual PCI card,
* in BAR 4.
* The module parameters take precedence
* over PCI memory.
*/
cfg->options.carveout.phys = contig_phys_start;
cfg->options.carveout.size = contig_size;
cfg->to_dev_addr = NULL;
cfg->to_host_addr = NULL;
dev_info(dev, "using %dMB CARVEOUT at x%lx\n",
contig_size/1024/1024,
contig_phys_start);
} else {
cfg->options.carveout.phys =
data->memmap[2].addr;
if (mem_static_kptr)
cfg->options.carveout.kptr =
data->memmap[2].km_addr;
cfg->options.carveout.size =
data->memmap[2].size;
cfg->options.carveout.offs = pci_offset;
cfg->to_dev_addr = carveout_to_dev_addr;
cfg->to_host_addr = carveout_to_host_addr;
dev_info(dev, "using %zuMB CARVEOUT from PCI at 0x%llx\n",
cfg->options.carveout.size/1024/1024,
cfg->options.carveout.phys);
}
/* IO memory access callbacks */
if (!mem_static_kptr) {
/* Dynamic kernel memory mapping */
cfg->options.carveout.get_kptr = carveout_get_kptr;
cfg->options.carveout.put_kptr = carveout_put_kptr;
}
break;
}
#endif
if (cfg->type == IMG_MEM_HEAP_TYPE_COHERENT) {
ret = dma_declare_coherent_memory(dev,
contig_phys_start, contig_phys_start,
contig_size
#if LINUX_VERSION_CODE < KERNEL_VERSION(5,1,0)
#if LINUX_VERSION_CODE < KERNEL_VERSION(4,14,0)
, DMA_MEMORY_MAP | DMA_MEMORY_EXCLUSIVE
#else
, DMA_MEMORY_EXCLUSIVE
#endif
#endif
);
if (ret == 0) {
dev_err(dev, "failed to initialize coherent memory!\n");
/*
* We will fallback to the default pool anyway
* goto out_release;
*/
}
break;
}
}
#ifdef FPGA_BUS_MASTERING
/* Allow the core driver to control pm_runtime */
pm_runtime_allow(dev);
#endif
ret = vha_add_dev(dev, vha_plat_fpga_heap_configs,
vha_plat_fpga_heaps, data,
data->memmap[1].km_addr, data->memmap[1].size);
if (ret) {
dev_err(dev, "failed to initialize driver core!\n");
goto out_heap_deinit;
}
/*
* Reset FPGA DUT only after disabling clocks in
* vha_add_dev()-> get properties.
* This workaround is required to ensure that
* clocks (on daughter board) are enabled for test slave scripts to
* read FPGA build version register.
* NOTE: Asserting other bits like DDR reset bit cause problems
* with bus mastering feature, thus results in memory failures.
*/
reset_fpga(pci_dev, data, PCI_ATLAS_DUT_RESET);
{
/*uint32_t fpga_rev = fpga_readreg32(data, 1,
FPGA_IMAGE_REV_OFFSET) & FPGA_IMAGE_REV_MASK;
dev_dbg(dev, "fpga image revision: 0x%x\n", fpga_rev);
if (!fpga_rev || fpga_rev == 0xdead1) {
dev_err(dev, "fpga revision incorrect (0x%x)!\n",
fpga_rev);
goto out_rm_dev;
}*/
}
/* Install the ISR callback...*/
ret = devm_request_threaded_irq(dev, data->irq, &pci_isrcb,
&pci_thread_irq, IRQF_SHARED, DEVICE_NAME,
(void *)pci_dev);
if (ret) {
dev_err(dev, "failed to request irq!\n");
goto out_rm_dev;
}
dev_dbg(dev, "registered irq %d\n", data->irq);
/* Try to calibrate the core if needed */
ret = vha_dev_calibrate(dev, FREQ_MEASURE_CYCLES);
if (ret) {
dev_err(dev, "%s: Failed to start clock calibration!\n", __func__);
goto out_rm_dev;
}
return ret;
out_rm_dev:
vha_rm_dev(dev);
out_heap_deinit:
#if LINUX_VERSION_CODE < KERNEL_VERSION(5,4,0)
/* Release any declared mem regions */
dma_release_declared_memory(dev);
#endif
/*out_release:*/
pci_release_regions(pci_dev);
out_disable:
pci_disable_device(pci_dev);
out_free:
return ret;
}
static void vha_plat_remove(struct pci_dev *dev)
{
struct imgpci_prvdata *data = vha_get_plat_data(&dev->dev);
dev_dbg(&dev->dev, "removing device\n");
if (data == NULL) {
dev_err(&dev->dev, "PCI priv data missing!\n");
} else {
/*
* We need to disable interrupts for the
* embedded device via the fpga interrupt controller...
*/
fpga_writereg32(data, PCI_ATLAS_SYS_CTRL_REGS_BAR,
interrupt_enable_reg, 0x00000000);
#ifdef FPGA_BUS_MASTERING
/* Route to internal RAM - this is reset value */
dev_dbg(&dev->dev, "disabling FPGA bus mastering\n");
fpga_writereg32(data,
PCI_ATLAS_SYS_CTRL_REGS_BAR,
test_ctrl_reg, 0x1);
#endif
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(5,4,0)
/* Release any declared mem regions */
dma_release_declared_memory(&dev->dev);
#endif
pci_release_regions(dev);
pci_disable_device(dev);
vha_rm_dev(&dev->dev);
#ifdef FPGA_BUS_MASTERING
pm_runtime_forbid(&dev->dev);
#endif
}
#ifdef CONFIG_PM
static int vha_plat_suspend(struct device *dev)
{
struct pci_dev *pci_dev = vha_pci_drv.pci_dev;
struct imgpci_prvdata *data = vha_get_plat_data(dev);
int ret;
ret = vha_suspend_dev(dev);
if (ret) {
dev_dbg(dev, "suspend device\n");
reset_fpga(pci_dev, data, PCI_ATLAS_DUT_RESET);
} else
dev_err(dev, "failed to suspend!\n");
return ret;
}
static int vha_plat_resume(struct device *dev)
{
struct pci_dev *pci_dev = vha_pci_drv.pci_dev;
struct imgpci_prvdata *data = vha_get_plat_data(dev);
int ret;
dev_dbg(dev, "resume device\n");
reset_fpga(pci_dev, data, PCI_ATLAS_DUT_RESET);
ret = vha_resume_dev(dev);
if (ret)
dev_err(dev, "failed to resume!\n");
return ret;
}
static int __maybe_unused vha_plat_runtime_idle(struct device *dev)
{
dev_dbg(dev, "%s\n", __func__);
return 0;
}
static int __maybe_unused vha_plat_runtime_suspend(struct device *dev)
{
dev_dbg(dev, "%s\n", __func__);
return 0;
}
static int __maybe_unused vha_plat_runtime_resume(struct device *dev)
{
dev_dbg(dev, "%s\n", __func__);
return 0;
}
#endif
int vha_plat_init(void)
{
int ret;
#ifdef FPGA_BUS_MASTERING
vha_plat_fpga_heap_configs[0].type = fpga_heap_type;
#endif
ret = pci_register_driver(&vha_pci_drv.pci_driver);
if (ret) {
pr_err("failed to register PCI driver!\n");
return ret;
}
/* pci_dev should be set in probe */
if (!vha_pci_drv.pci_dev) {
pr_err("failed to find VHA PCI dev!\n");
pci_unregister_driver(&vha_pci_drv.pci_driver);
return -ENODEV;
}
return 0;
}

386
drivers/nna/vha/platform/vha_plat_dt.c Normal file
View File

@@ -0,0 +1,386 @@
/*!
*****************************************************************************
*
* @File vha_plat_dt.c
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/interrupt.h>
#include <linux/device.h>
#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/io.h>
#include <linux/pm.h>
#include <linux/version.h>
#include <img_mem_man.h>
#include "vha_common.h"
#include "uapi/version.h"
#include "vha_plat.h"
#include "vha_plat_dt.h"
#if defined(CFG_SYS_VAGUS)
#include <hwdefs/vagus_system.h>
#elif defined(CFG_SYS_AURA)
#include <hwdefs/aura_system.h>
#elif defined(CFG_SYS_MIRAGE)
#include <hwdefs/mirage_system.h>
#endif
#define DEVICE_NAME "vha"
/* Number of core cycles used to measure the core clock frequency */
#define FREQ_MEASURE_CYCLES 0xfffffff
static bool poll_interrupts; /* Disabled by default */
module_param(poll_interrupts, bool, 0444);
MODULE_PARM_DESC(poll_interrupts, "Poll for interrupts? 0: No, 1: Yes");
static unsigned int irq_poll_interval_ms = 100; /* 100 ms */
module_param(irq_poll_interval_ms, uint, 0444);
MODULE_PARM_DESC(irq_poll_interval_ms, "Time in ms between each interrupt poll");
/* Global timer used when irq poll mode is switched on.
* NOTE: only single core instance is supported in polling mode */
static struct poll_timer {
struct platform_device *pdev;
struct timer_list tmr;
bool enabled;
} irq_poll_timer;
static irqreturn_t dt_plat_thread_irq(int irq, void *dev_id)
{
struct platform_device *ofdev = (struct platform_device *)dev_id;
return vha_handle_thread_irq(&ofdev->dev);
}
static irqreturn_t dt_plat_isrcb(int irq, void *dev_id)
{
struct platform_device *ofdev = (struct platform_device *)dev_id;
if (!ofdev)
return IRQ_NONE;
return vha_handle_irq(&ofdev->dev);
}
/* Interrupt polling function */
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,15,0)
static void dt_plat_poll_interrupt(struct timer_list *t)
{
struct poll_timer *poll_timer = from_timer(poll_timer, t, tmr);
#else
static void dt_plat_poll_interrupt(unsigned long ctx)
{
struct poll_timer *poll_timer = (struct poll_timer *)ctx;
#endif
struct platform_device *ofdev = poll_timer->pdev;
int ret;
if (!poll_timer->enabled)
return;
preempt_disable();
ret = vha_handle_irq(&ofdev->dev);
preempt_enable();
if (ret == IRQ_WAKE_THREAD)
vha_handle_thread_irq(&ofdev->dev);
/* retrigger */
mod_timer(&poll_timer->tmr,
jiffies + msecs_to_jiffies(irq_poll_interval_ms));
}
static int vha_plat_probe(struct platform_device *ofdev)
{
int ret, module_irq;
struct resource res;
void __iomem *reg_addr;
uint32_t reg_size, core_size;
ret = of_address_to_resource(ofdev->dev.of_node, 0, &res);
if (ret) {
dev_err(&ofdev->dev, "missing 'reg' property in device tree\n");
return ret;
}
pr_info("%s: registers %#llx-%#llx\n", __func__,
(unsigned long long)res.start, (unsigned long long)res.end);
module_irq = irq_of_parse_and_map(ofdev->dev.of_node, 0);
if (module_irq == 0) {
dev_err(&ofdev->dev, "could not map IRQ\n");
return -ENXIO;
}
/* Assuming DT holds a single registers space entry that covers all regions,
* So we need to do the split accordingly */
reg_size = res.end - res.start + 1;
#ifdef CFG_SYS_VAGUS
core_size = _REG_SIZE + _REG_NNSYS_SIZE;
#else
core_size = _REG_SIZE;
#endif
if ((res.start + _REG_START) > res.end) {
dev_err(&ofdev->dev, "wrong system conf for core region!\n");
return -ENXIO;
}
if ((res.start + _REG_START + core_size) > res.end) {
dev_warn(&ofdev->dev, "trimming system conf for core region!\n");
core_size = reg_size - _REG_START;
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
reg_addr = devm_ioremap_nocache(&ofdev->dev, res.start +
_REG_START, core_size);
#else
reg_addr = devm_ioremap(&ofdev->dev, res.start +
_REG_START, core_size);
#endif
if (!reg_addr) {
dev_err(&ofdev->dev, "failed to map core registers\n");
return -ENXIO;
}
ret = vha_plat_dt_hw_init(ofdev);
if (ret) {
dev_err(&ofdev->dev, "failed to init platform-specific hw!\n");
goto out_add_dev;
}
/* no 'per device' memory heaps used */
ret = vha_add_dev(&ofdev->dev, NULL, 0,
NULL /* plat priv data */, reg_addr, core_size);
if (ret) {
dev_err(&ofdev->dev, "failed to intialize driver core!\n");
goto out_add_dev;
}
if (!poll_interrupts) {
ret = devm_request_threaded_irq(&ofdev->dev, module_irq, &dt_plat_isrcb,
&dt_plat_thread_irq, IRQF_SHARED, DEVICE_NAME, ofdev);
if (ret) {
dev_err(&ofdev->dev, "failed to request irq\n");
goto out_irq;
}
} else {
irq_poll_timer.pdev = ofdev;
irq_poll_timer.enabled = true;
/* Setup and start poll timer */
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,15,0)
timer_setup(&irq_poll_timer.tmr, dt_plat_poll_interrupt, 0);
#else
setup_timer(&irq_poll_timer.tmr, dt_plat_poll_interrupt,
(uintptr_t)&irq_poll_timer);
#endif
mod_timer(&irq_poll_timer.tmr,
jiffies + msecs_to_jiffies(irq_poll_interval_ms));
}
/* Try to calibrate the core if needed */
ret = vha_dev_calibrate(&ofdev->dev, FREQ_MEASURE_CYCLES);
if (ret) {
dev_err(&ofdev->dev, "%s: Failed to start clock calibration!\n", __func__);
goto out_irq;
}
return ret;
out_irq:
vha_rm_dev(&ofdev->dev);
out_add_dev:
devm_iounmap(&ofdev->dev, reg_addr);
return ret;
}
static int vha_plat_remove(struct platform_device *ofdev)
{
vha_rm_dev(&ofdev->dev);
vha_plat_dt_hw_destroy(ofdev);
return 0;
}
#ifdef CONFIG_PM
static int vha_plat_suspend(struct device *dev)
{
struct platform_device *ofdev =
container_of(dev, struct platform_device, dev);
int ret = 0;
ret = vha_suspend_dev(dev);
if (ret)
dev_err(dev, "failed to suspend the core!\n");
else {
ret = vha_plat_dt_hw_suspend(ofdev);
if (ret)
dev_err(dev, "failed to suspend platform-specific hw!\n");
}
return ret;
}
static int vha_plat_resume(struct device *dev)
{
struct platform_device *ofdev =
container_of(dev, struct platform_device, dev);
int ret = 0;
ret = vha_plat_dt_hw_resume(ofdev);
if (ret)
dev_err(dev, "failed to resume platform-specific hw!\n");
else {
ret = vha_resume_dev(dev);
if (ret)
dev_err(dev, "failed to resume the core!\n");
}
return ret;
}
static int vha_plat_runtime_idle(struct device *dev)
{
/* Eg. turn off external clocks */
return 0;
}
static int vha_plat_runtime_suspend(struct device *dev)
{
/* Nothing to do */
return 0;
}
static int vha_plat_runtime_resume(struct device *dev)
{
/* Eg. turn on external clocks */
return 0;
}
#endif
static struct dev_pm_ops vha_pm_plat_ops = {
SET_RUNTIME_PM_OPS(vha_plat_runtime_suspend,
vha_plat_runtime_resume, vha_plat_runtime_idle)
SET_SYSTEM_SLEEP_PM_OPS(vha_plat_suspend, vha_plat_resume)
};
static ssize_t info_show(struct device_driver *drv, char *buf)
{
return sprintf(buf, "VHA DT driver version : " VERSION_STRING "\n");
}
static DRIVER_ATTR_RO(info);
static struct attribute *drv_attrs[] = {
&driver_attr_info.attr,
NULL
};
ATTRIBUTE_GROUPS(drv);
static struct platform_driver vha_plat_drv = {
.probe = vha_plat_probe,
.remove = vha_plat_remove,
.driver = {
.name = VHA_PLAT_DT_NAME,
.groups = drv_groups,
.owner = THIS_MODULE,
.of_match_table = vha_plat_dt_of_ids,
.pm = &vha_pm_plat_ops,
},
};
int vha_plat_init(void)
{
int ret = 0;
struct heap_config *heap_configs;
int num_heaps;
vha_plat_dt_get_heaps(&heap_configs, &num_heaps);
ret = vha_init_plat_heaps(heap_configs, num_heaps);
if (ret) {
pr_err("failed to initialize global heaps\n");
return -ENOMEM;
}
ret = platform_driver_register(&vha_plat_drv);
if (ret) {
pr_err("failed to register VHA driver!\n");
return ret;
}
return 0;
}
int vha_plat_deinit(void)
{
int ret;
if (poll_interrupts) {
irq_poll_timer.enabled = false;
del_timer_sync(&irq_poll_timer.tmr);
}
/* Unregister the driver from the OS */
platform_driver_unregister(&vha_plat_drv);
ret = vha_deinit();
if (ret)
pr_err("VHA driver deinit failed\n");
return ret;
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

78
drivers/nna/vha/platform/vha_plat_dt.h Normal file
View File

@@ -0,0 +1,78 @@
/*!
*****************************************************************************
*
* @File vha_plat_dt.h
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#ifndef VHA_PLAT_DT_H
#define VHA_PLAT_DT_H
#include <linux/platform_device.h>
/* OpenFirmware device tree id, for this driver */
#if defined(HW_AX2)
#define VHA_PLAT_DT_OF_ID "img,ax21xx-nna"
#define VHA_PLAT_DT_NAME "ax21xx-nna"
#elif defined(HW_AX3)
#define VHA_PLAT_DT_OF_ID "img,ax3xxx-nna"
#define VHA_PLAT_DT_NAME "ax3xxx-nna"
#else
#error "No HW layout defined"
#endif
extern const struct of_device_id vha_plat_dt_of_ids[];
void vha_plat_dt_get_heaps(struct heap_config **heap_configs, int *num_heaps);
int vha_plat_dt_hw_init(struct platform_device *pdev);
void vha_plat_dt_hw_destroy(struct platform_device *pdev);
int vha_plat_dt_hw_suspend(struct platform_device *pdev);
int vha_plat_dt_hw_resume(struct platform_device *pdev);
#endif /* VHA_PLAT_DT_H */

156
drivers/nna/vha/platform/vha_plat_dt_example.c Normal file
View File

@@ -0,0 +1,156 @@
/*!
*****************************************************************************
*
* @File vha_plat_dt_example.c
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/module.h>
#include <linux/gfp.h>
#include <linux/device.h>
#include <linux/dma-mapping.h>
#include <linux/of.h>
#include <img_mem_man.h>
#include "vha_plat.h"
#include "vha_plat_dt.h"
const struct of_device_id vha_plat_dt_of_ids[] = {
{ .compatible = VHA_PLAT_DT_OF_ID },
{ }
};
static struct heap_config example_heap_configs[] = {
{
.type = IMG_MEM_HEAP_TYPE_UNIFIED,
.options.unified = {
.gfp_type = GFP_KERNEL | __GFP_ZERO,
},
.to_dev_addr = NULL,
},
{
.type = IMG_MEM_HEAP_TYPE_DMABUF,
.to_dev_addr = NULL,
},
{
.type = IMG_MEM_HEAP_TYPE_ANONYMOUS,
.to_dev_addr = NULL,
},
};
/*
* IO hooks.
* NOTE: customer may want to use spinlock to avoid
* problems with multi threaded IO access
*/
uint64_t vha_plat_read64(void *addr)
{
return readq((volatile void __iomem *)addr);
}
void vha_plat_write64(void *addr, uint64_t val)
{
writeq(val, (volatile void __iomem *)addr);
}
int vha_plat_dt_hw_init(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
int ret;
uint64_t dma_mask;
dev_dbg(dev, "%s dma_get_mask : %#llx\n", __func__, dma_get_mask(dev));
if (dev->dma_mask) {
dev_info(dev, "%s dev->dma_mask : %p : %#llx\n",
__func__, dev->dma_mask, *dev->dma_mask);
} else {
dev_info(dev, "%s mask unset, setting coherent\n", __func__);
dev->dma_mask = &dev->coherent_dma_mask;
}
/* Try alternative dma_mask setting from device tree */
if (!of_property_read_u64(pdev->dev.of_node, "dma-mask",
(uint64_t *)&dma_mask)) {
dev_info(dev, "%s forcing custom mask from DT : %#llx\n",
__func__, dma_mask);
} else {
/* If alternative mask not defined in
* DT -> "dma-mask" property, use the default one (32bit) */
dma_mask = dma_get_mask(dev);
}
ret = dma_set_mask(dev, dma_mask);
if (ret) {
dev_err(dev, "%s failed to set dma mask\n", __func__);
return ret;
}
/* Put any vendor related code:
* get clock domain, voltage regulator, set clock rate, etc */
return 0;
}
/* return platform global heaps */
void vha_plat_dt_get_heaps(struct heap_config **heap_configs, int *num_heaps)
{
*heap_configs = example_heap_configs;
*num_heaps = sizeof(example_heap_configs)/sizeof(struct heap_config);
}
void vha_plat_dt_hw_destroy(struct platform_device *pdev)
{
/* Put any vendor related code:
* put clock domain, voltage regulator, etc */
}
int vha_plat_dt_hw_suspend(struct platform_device *pdev)
{
/* This is the place where vendor specific code shall be called:
* eg. turn off voltage regulator/disable power domain */
return 0;
}
int vha_plat_dt_hw_resume(struct platform_device *pdev)
{
/* This is the place where vendor specific code shall be called:
* eg. turn on voltage regulator/enable power domain */
return 0;
}
MODULE_DEVICE_TABLE(of, vha_plat_dt_of_ids);

60
drivers/nna/vha/platform/vha_plat_dt_example.dts Normal file
View File

@@ -0,0 +1,60 @@
/*!
*****************************************************************************
*
* @File vha_plat_dt_example.dts
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
/ {
/* ... */
vha {
compatible = "img,ax21xx-nna";
reg = <0x0 0xe8800000 0x0 0x100000>;
interrupts = <0 257 4>;
clocks = <&clk_gate_vha>;
clock-names = "clk_vha";
vha_clk_rate = <800000000>;
ldo_vha-supply = <&vha>;
dma-mask = 0xffffffffff; /* 40bit mask */
status = enabled;
};
/* ... */
};

81
drivers/nna/vha/platform/vha_plat_dt_fenrir.dts Normal file
View File

@@ -0,0 +1,81 @@
/*!
*****************************************************************************
*
* @File vha_plat_dt_fenrir.dts
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
/* Build this file using:
*
* dtc -@ -I dts -O dtb vha_plat_dt_fenrir.dts -o vha_plat_dt_fenrir.dtbo
*
* The loading process on the target is done this way:
*
* sudo mkdir /sys/kernel/config/device-tree/overlays/nna
* cat vha_plat_dt_fenrir.dtbo | sudo tee /sys/kernel/config/device-tree/overlays/nna/dtbo > /dev/null
*
* This will apply the device overlay and set the needed "compatible" entry for the driver to load.
*/
/dts-v1/;
/plugin/;
&m_loki_core {
compatible = "img,loki";
interrupt-parent = <&gic>;
interrupts = <0x0 0x59 0x4>;
interrupt-controller;
#interrupt-cells = <1>;
/* Not needed at the moment, but keep them just in case */
//memif-cache = <0x0>;
//memif-prot = <0x0>;
};
&m_dut_socif {
compatible = "img,ax3xxx-nna";
interrupt-parent = <&m_loki_core>;
interrupts = <0x0>;
dma-mask = /bits/ 64 <0xFFFFFFFFF>;
};

361
drivers/nna/vha/platform/vha_plat_dummy.c Normal file
View File

@@ -0,0 +1,361 @@
/*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/module.h>
#include <linux/device.h>
#include <linux/gfp.h>
#include <linux/dma-mapping.h>
#include <linux/pci.h>
#include <linux/pm.h>
#include <linux/mod_devicetable.h>
#include <linux/platform_device.h>
#include <linux/moduleparam.h>
#include <linux/vmalloc.h>
#include "vha_common.h"
#include "vha_plat.h"
#include "uapi/version.h"
#include "vha_regs.h"
#if defined(CFG_SYS_MAGNA)
#include "hwdefs/vha_cr_magna.h"
#endif
#define DEVICE_NAME "vha"
static unsigned short min_alloc_order;
module_param(min_alloc_order, ushort, 0444);
MODULE_PARM_DESC(min_alloc_order,
"Minimum allocation order, depends on PAGE_SIZE, \
for CPU PAGE_SIZE=4kB, 0-4kB, 1-8kB, 2-16kB, 3-32kB, 4-64kB");
static unsigned short max_alloc_order = 10; /* 4MB for PAGE_SIZE=4kB */
module_param(max_alloc_order, ushort, 0444);
MODULE_PARM_DESC(max_alloc_order,
"Maximum allocation order, depends on PAGE_SIZE, \
for CPU PAGE_SIZE=4kB, 0-4kB, 1-8kB, 2-16kB, 3-32kB, 4-64kB");
static unsigned char num_clusters = 1;
module_param(num_clusters, byte, 0444);
MODULE_PARM_DESC(num_clusters,
"Number of dummy clusters. Each cluster will be instantiated "
"as separate /dev/vhaN node. Max number supported is 255.");
static struct heap_config dummy_heap_configs[] = {
/* the first config is default */
{
.type = IMG_MEM_HEAP_TYPE_UNIFIED,
.options.unified = {
.gfp_type = GFP_KERNEL | __GFP_ZERO,
},
.to_dev_addr = NULL,
},
{
.type = IMG_MEM_HEAP_TYPE_ANONYMOUS,
},
#ifdef CONFIG_DMA_SHARED_BUFFER
{
.type = IMG_MEM_HEAP_TYPE_DMABUF,
.to_dev_addr = NULL,
},
#endif
};
static size_t num_dummy_heaps =
sizeof(dummy_heap_configs) / sizeof(*dummy_heap_configs);
static const size_t nna_regs_size =
#ifdef _REG_NNSYS_SIZE
_REG_NNSYS_SIZE +
#endif
_REG_SIZE;
static bool use_dummy_regs = false;
/* IO hooks - do nothing */
uint64_t vha_plat_read64(void *addr)
{
if (use_dummy_regs)
return *((uint64_t*)addr);
return 0ULL;
}
void vha_plat_write64(void *addr, uint64_t val)
{
if (use_dummy_regs)
*((uint64_t*)addr) = val;
}
#if defined(CFG_SYS_MAGNA)
static void vha_plat_magna_write_defaults(uint8_t* nna_regs) {
if (!use_dummy_regs)
return;
*((uint64_t*)(nna_regs + VHA_CR_CORE_ASSIGNMENT)) =
VHA_CR_CORE_ASSIGNMENT_CORE_7_WM_MAPPING_UNALLOCATED |
VHA_CR_CORE_ASSIGNMENT_CORE_6_WM_MAPPING_UNALLOCATED |
VHA_CR_CORE_ASSIGNMENT_CORE_5_WM_MAPPING_UNALLOCATED |
VHA_CR_CORE_ASSIGNMENT_CORE_4_WM_MAPPING_UNALLOCATED |
VHA_CR_CORE_ASSIGNMENT_CORE_3_WM_MAPPING_UNALLOCATED |
VHA_CR_CORE_ASSIGNMENT_CORE_2_WM_MAPPING_UNALLOCATED |
VHA_CR_CORE_ASSIGNMENT_CORE_1_WM_MAPPING_UNALLOCATED |
VHA_CR_CORE_ASSIGNMENT_CORE_0_WM_MAPPING_UNALLOCATED;
*((uint64_t*)(nna_regs + VHA_CR_SOCM_BUF_ASSIGNMENT)) =
VHA_CR_SOCM_BUF_ASSIGNMENT_SOCM_BUF_7_WM_MAPPING_UNALLOCATED |
VHA_CR_SOCM_BUF_ASSIGNMENT_SOCM_BUF_6_WM_MAPPING_UNALLOCATED |
VHA_CR_SOCM_BUF_ASSIGNMENT_SOCM_BUF_5_WM_MAPPING_UNALLOCATED |
VHA_CR_SOCM_BUF_ASSIGNMENT_SOCM_BUF_4_WM_MAPPING_UNALLOCATED |
VHA_CR_SOCM_BUF_ASSIGNMENT_SOCM_BUF_3_WM_MAPPING_UNALLOCATED |
VHA_CR_SOCM_BUF_ASSIGNMENT_SOCM_BUF_2_WM_MAPPING_UNALLOCATED |
VHA_CR_SOCM_BUF_ASSIGNMENT_SOCM_BUF_1_WM_MAPPING_UNALLOCATED |
VHA_CR_SOCM_BUF_ASSIGNMENT_SOCM_BUF_0_WM_MAPPING_UNALLOCATED;
}
#endif
#ifdef CONFIG_PM
static int vha_plat_suspend(struct device *dev)
{
return vha_suspend_dev(dev);
}
static int vha_plat_resume(struct device *dev)
{
return vha_resume_dev(dev);
}
static int vha_plat_runtime_idle(struct device *dev)
{
dev_dbg(dev, "%s\n", __func__);
return 0;
}
static int vha_plat_runtime_suspend(struct device *dev)
{
dev_dbg(dev, "%s\n", __func__);
return 0;
}
static int vha_plat_runtime_resume(struct device *dev)
{
dev_dbg(dev, "%s\n", __func__);
return 0;
}
#endif
static int vha_plat_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
void* nna_regs=NULL;
int ret = 0;
/* dma_mask is required in order for dma_ops mapping to work */
dev_info(dev, "%s dma_get_mask : %#llx\n", __func__, dma_get_mask(dev));
if (dev->dma_mask) {
dev_info(dev, "%s dev->dma_mask : %p : %#llx\n",
__func__, dev->dma_mask, *dev->dma_mask);
} else {
dev_info(dev, "%s mask unset, setting coherent\n", __func__);
dev->dma_mask = &dev->coherent_dma_mask;
}
/* Give 128GB fake dma address range,
* so that dma_map_page/map_sg does not throw any error,
* when dealing with high mem address alocations */
ret = dma_set_mask(dev, DMA_BIT_MASK(37));
if (ret) {
dev_err(dev, "%s failed to set dma mask\n", __func__);
goto out_add_dev;
}
dev_info(dev, "%s dma_set_mask %#llx\n",
__func__, dma_get_mask(dev));
if (dev->platform_data)
nna_regs = *(uint8_t**)dev->platform_data;
ret = vha_add_dev(dev, NULL, 0,
NULL /* plat data */,
nna_regs /* reg base */,
nna_regs_size /* reg size*/);
if (ret) {
dev_err(dev, "vha_add_dev failed\n");
goto out_add_dev;
}
out_add_dev:
return ret;
}
static int vha_plat_remove(struct platform_device *pdev)
{
int ret = 0;
dev_info(&pdev->dev, "%s\n", __func__);
vha_rm_dev(&pdev->dev);
return ret;
}
static struct dev_pm_ops vha_pm_plat_ops = {
SET_RUNTIME_PM_OPS(vha_plat_runtime_suspend,
vha_plat_runtime_resume, vha_plat_runtime_idle)
SET_SYSTEM_SLEEP_PM_OPS(vha_plat_suspend, vha_plat_resume)
};
static ssize_t info_show(struct device_driver *drv, char *buf)
{
return sprintf(buf, "VHA dummy driver version : " VERSION_STRING "\n");
}
static DRIVER_ATTR_RO(info);
static struct attribute *drv_attrs[] = {
&driver_attr_info.attr,
NULL
};
ATTRIBUTE_GROUPS(drv);
static struct platform_driver vha_driver = {
.driver = {
.owner = THIS_MODULE,
.name = "vha",
.groups = drv_groups,
.pm = &vha_pm_plat_ops,
},
.probe = vha_plat_probe,
.remove = vha_plat_remove,
};
static struct platform_device **pd;
int __exit vha_plat_deinit(void)
{
int ret;
int cluster;
uint8_t* nna_regs;
platform_driver_unregister(&vha_driver);
for (cluster=0; cluster<num_clusters; ++cluster) {
BUG_ON(pd[cluster]==NULL);
nna_regs = *(uint8_t**)pd[cluster]->dev.platform_data;
vfree(nna_regs);
platform_device_unregister(pd[cluster]);
}
use_dummy_regs = false;
kfree(pd);
ret = vha_deinit();
if (ret)
pr_err("VHA driver deinit failed\n");
return 0;
}
int __init vha_plat_init(void)
{
int ret;
int cluster;
if (min_alloc_order > max_alloc_order) {
pr_err("Can't set min_alloc_order > max_alloc_order !\n");
return -EINVAL;
}
WARN_ON(dummy_heap_configs[0].type != IMG_MEM_HEAP_TYPE_UNIFIED);
dummy_heap_configs[0].options.unified.min_order = min_alloc_order;
dummy_heap_configs[0].options.unified.max_order = max_alloc_order;
ret = vha_init_plat_heaps(dummy_heap_configs, num_dummy_heaps);
if (ret) {
pr_err("failed to initialize global heaps\n");
return -ENOMEM;
}
if (num_clusters == 0) {
pr_notice("Overriding num_clusters parameter to 1\n");
num_clusters=1;
}
pr_notice("%s instantiating %d dummy clusters\n",
__func__, num_clusters);
pd = kmalloc(num_clusters*sizeof(*pd), GFP_KERNEL);
if (pd == NULL) {
pr_err("failed to allocate memory!\n");
return -ENOMEM;
}
memset(pd, 0, num_clusters*sizeof(*pd));
for (cluster=0; cluster<num_clusters; ++cluster) {
void* nna_regs=NULL;
pr_notice("%s Instantiating dummy vha%d cluster\n", __func__, cluster);
#ifdef _REG_NNA_SIZE
nna_regs = vmalloc(nna_regs_size);
if (nna_regs == NULL)
pr_warn("Failed allocating dummy NNA reg space. Will not use it...\n");
else {
pr_notice("Successfully allocated dummy NNA reg space.\n");
memset(nna_regs, 0, nna_regs_size);
use_dummy_regs = true;
#if defined(CFG_SYS_MAGNA)
vha_plat_magna_write_defaults(nna_regs);
#endif
}
#endif
/* after this call a copy of pointer to nna_regs is stored in struct device.platform_data
* internal data is managed by platform device */
pd[cluster] = platform_device_register_data(NULL, "vha", cluster, &nna_regs, sizeof(&nna_regs));
ret = IS_ERR(pd[cluster]);
if (ret) {
pr_err("failed to register platform device!\n");
goto _err;
}
}
ret = platform_driver_register(&vha_driver);
if (ret) {
pr_err("failed to register platform driver!\n");
goto _err;
}
return ret;
_err:
for (cluster=0; cluster<num_clusters; ++cluster) {
uint8_t* nna_regs = *(uint8_t**)pd[cluster]->dev.platform_data;
vfree(nna_regs);
platform_device_unregister(pd[cluster]);
}
kfree(pd);
return ret;
}

641
drivers/nna/vha/platform/vha_plat_emu.c Normal file
View File

@@ -0,0 +1,641 @@
/*!
*****************************************************************************
*
* @File vha_plat_emu.c
* ---------------------------------------------------------------------------
*
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/module.h>
#include <linux/device.h>
#include <linux/gfp.h>
#include <linux/dma-mapping.h>
#include <linux/pci.h>
#include <linux/pm.h>
#include <linux/mod_devicetable.h>
#include "uapi/version.h"
#include "vha_common.h"
#include "vha_plat.h"
#define DEVICE_NAME "vha"
/*
* Spec:
* Emulator PCIe In-Circuit Interface Card.Technical
* Reference Manual.1.0.24.External PSTDRW.External
*/
/* Emulator address range 0x4000-0x4FFF */
#define PCI_EMU_SYS_CTRL_REGS_BAR (0)
/* Offset of INTERRUPT_ENABLE */
#define PCI_EMU_INTERRUPT_ENABLE_OFS (0x4048)
/* master interrupt enable - default high */
#define PCI_EMU_IRQ_ENABLE (1<<0)
#define PCI_EMU_IRQ_HIGH (1<<1)
/* Emulator reset offset */
#define PCI_EMU_RESET_OFS (0x4000)
/* Emulator reset bits */
#define PCI_EMU_RESET_LOGIC (1<<0)
#define PCI_EMU_RESET_DUT (1<<1)
#define PCI_EMU_VENDOR_ID (0x1010)
#define PCI_EMU_DEVICE_ID (0x1CE3)
#define NUM_EMU_BARS 3
#define EMU_REG_BAR PCI_EMU_SYS_CTRL_REGS_BAR
#define NNA_REG_BAR 1
#define NNA_MEM_BAR 2
/* Number of core cycles used to measure the core clock frequency */
#define FREQ_MEASURE_CYCLES 0x7fffff
static unsigned long pci_size;
module_param(pci_size, ulong, 0444);
MODULE_PARM_DESC(pci_size, "physical size in bytes. when 0 (the default), use all memory in the PCI bar");
static unsigned long pci_offset;
module_param(pci_offset, ulong, 0444);
MODULE_PARM_DESC(pci_offset, "offset from PCI bar start. (default: 0)");
static unsigned short pool_alloc_order;
module_param(pool_alloc_order, ushort, 0444);
MODULE_PARM_DESC(pool_alloc_order,
"Carveout pool allocation order, depends on PAGE_SIZE, \
for CPU PAGE_SIZE=4kB, 0-4kB, 1-8kB, 2-16kB, 3-32kB, 4-64kB");
static unsigned long poll_interrupts = 1; /* Enabled by default */
module_param(poll_interrupts, ulong, 0444);
MODULE_PARM_DESC(poll_interrupts, "Poll for interrupts? 0: No, 1: Yes");
static unsigned long irq_poll_delay_us = 10000; /* 10 ms */
module_param(irq_poll_delay_us, ulong, 0444);
MODULE_PARM_DESC(irq_poll_delay_us, "Delay in us between each interrupt poll");
static bool mem_static_kptr = true;
module_param(mem_static_kptr, bool, 0444);
MODULE_PARM_DESC(mem_static_kptr,
"Creates static kernel mapping for fpga memory");
static struct heap_config vha_plat_emu_heap_configs[] = {
#ifdef CONFIG_GENERIC_ALLOCATOR
{
.type = IMG_MEM_HEAP_TYPE_CARVEOUT,
/* .options.carveout to be filled at run time */
/* .to_dev_addr to be filled at run time */
/* .to_host_addr to be filled at run time */
},
#else
#error CONFIG_GENERIC_ALLOCATOR was not defined
#endif
#if CONFIG_DMA_SHARED_BUFFER
{
.type = IMG_MEM_HEAP_TYPE_DMABUF,
.to_dev_addr = NULL,
.options.dmabuf = {
.use_sg_dma = true,
},
},
#else
#warning "Memory importing not supported!"
#endif
};
static const int vha_plat_emu_heaps =
sizeof(vha_plat_emu_heap_configs)/sizeof(*vha_plat_emu_heap_configs);
static const struct pci_device_id pci_pci_ids[] = {
{ PCI_DEVICE(PCI_EMU_VENDOR_ID, PCI_EMU_DEVICE_ID) },
{ 0, }
};
MODULE_DEVICE_TABLE(pci, pci_pci_ids);
struct imgpci_prvdata {
int irq;
struct {
unsigned long addr;
unsigned long size;
void __iomem *km_addr;
} memmap[NUM_EMU_BARS];
struct pci_dev *pci_dev;
int irq_poll;
struct delayed_work irq_work;
};
struct img_pci_driver {
struct pci_dev *pci_dev;
struct pci_driver pci_driver;
};
static int vha_plat_probe(struct pci_dev *pci_dev,
const struct pci_device_id *id);
static void vha_plat_remove(struct pci_dev *dev);
static int vha_plat_suspend(struct device *dev);
static int vha_plat_resume(struct device *dev);
static SIMPLE_DEV_PM_OPS(vha_pm_plat_ops,
vha_plat_suspend, vha_plat_resume);
static ssize_t info_show(struct device_driver *drv, char *buf)
{
return sprintf(buf, "VHA EMU driver version : " VERSION_STRING "\n");
}
static DRIVER_ATTR_RO(info);
static struct attribute *drv_attrs[] = {
&driver_attr_info.attr,
NULL
};
ATTRIBUTE_GROUPS(drv);
static struct img_pci_driver vha_pci_drv = {
.pci_driver = {
.name = "vha_pci",
.id_table = pci_pci_ids,
.probe = vha_plat_probe,
.remove = vha_plat_remove,
.driver = {
.groups = drv_groups,
.pm = &vha_pm_plat_ops,
}
},
};
static ulong maxmapsizeMB = (sizeof(void *) == 4) ? 400 : 1024;
#if 0
static unsigned int emu_readreg32(struct imgpci_prvdata *data,
int bar, unsigned long offset
)
{
void __iomem *reg =
(void __iomem *)(data->memmap[bar].km_addr + offset);
return ioread32(reg);
}
#endif
static void emu_writereg32(struct imgpci_prvdata *data,
int bar, unsigned long offset, int val)
{
void __iomem *reg =
(void __iomem *)(data->memmap[bar].km_addr + offset);
iowrite32(val, reg);
}
static void reset_emu(struct pci_dev *dev,
struct imgpci_prvdata *data)
{
if (!dev)
return;
emu_writereg32(data, PCI_EMU_SYS_CTRL_REGS_BAR,
PCI_EMU_RESET_OFS,
~(PCI_EMU_RESET_LOGIC|PCI_EMU_RESET_DUT));
mdelay(100);
emu_writereg32(data, PCI_EMU_SYS_CTRL_REGS_BAR,
PCI_EMU_RESET_OFS,
PCI_EMU_RESET_LOGIC|PCI_EMU_RESET_DUT);
}
static irqreturn_t pci_thread_irq(int irq, void *dev_id)
{
struct pci_dev *dev = (struct pci_dev *)dev_id;
return vha_handle_thread_irq(&dev->dev);
}
static irqreturn_t pci_handle_irq(int irq, void *dev_id)
{
struct pci_dev *dev = (struct pci_dev *)dev_id;
struct imgpci_prvdata *data = vha_get_plat_data(&dev->dev);
irqreturn_t ret = IRQ_NONE;
if (data == NULL || dev_id == NULL) {
/* spurious interrupt: not yet initialised. */
goto exit;
}
ret = vha_handle_irq(&dev->dev);
exit:
return ret;
}
/* Interrupt polling function */
static void pci_poll_interrupt(struct work_struct *work)
{
struct imgpci_prvdata *data = container_of(work,
struct imgpci_prvdata, irq_work.work);
struct pci_dev *dev = data->pci_dev;
int ret;
if (!data->irq_poll)
return;
preempt_disable();
ret = vha_handle_irq(&dev->dev);
preempt_enable();
if (ret == IRQ_WAKE_THREAD)
vha_handle_thread_irq(&dev->dev);
/* retrigger */
schedule_delayed_work(&data->irq_work,
usecs_to_jiffies(irq_poll_delay_us));
}
/*
* IO hooks.
* NOTE: customer may want to use spinlock to avoid
* problems with multi threaded IO access
*/
static DEFINE_SPINLOCK(io_irq_lock);
static unsigned long io_irq_flags;
uint64_t vha_plat_read64(void *addr)
{
u64 val;
spin_lock_irqsave(&io_irq_lock, io_irq_flags);
val =(uint64_t)readl((const volatile void __iomem *)addr) |
((uint64_t)readl((const volatile void __iomem *)addr + 4) << 32);
spin_unlock_irqrestore(&io_irq_lock, io_irq_flags);
return val;
}
void vha_plat_write64(void *addr, uint64_t val)
{
spin_lock_irqsave(&io_irq_lock, io_irq_flags);
writel((uint32_t)(val & 0xffffffff), (volatile void __iomem *)addr);
writel((uint32_t)(val >> 32), (volatile void __iomem *)addr + 4);
spin_unlock_irqrestore(&io_irq_lock, io_irq_flags);
}
int vha_plat_deinit(void)
{
struct pci_dev *dev = vha_pci_drv.pci_dev;
int ret;
if (dev) {
struct imgpci_prvdata *data = vha_get_plat_data(&dev->dev);
if (data) {
if (poll_interrupts) {
data->irq_poll = 0;
cancel_delayed_work_sync(&data->irq_work);
}
/* reset the emulator */
reset_emu(data->pci_dev, data);
emu_writereg32(data,
PCI_EMU_SYS_CTRL_REGS_BAR,
PCI_EMU_INTERRUPT_ENABLE_OFS,
~PCI_EMU_IRQ_ENABLE);
} else {
dev_dbg(&dev->dev,
"%s: prv data not found, HW reset omitted\n",
__func__);
}
} else {
pr_debug("%s: dev missing, HW reset omitted\n", __func__);
}
/* Unregister the driver from the OS */
pci_unregister_driver(&(vha_pci_drv.pci_driver));
ret = vha_deinit();
if (ret)
pr_err("VHA driver deinit failed\n");
return ret;
}
#ifdef CONFIG_GENERIC_ALLOCATOR
static phys_addr_t carveout_to_dev_addr(union heap_options *options,
phys_addr_t addr)
{
phys_addr_t base = options->carveout.phys;
size_t size = options->carveout.size;
unsigned long offset = options->carveout.offs;
if (addr - offset >= base && addr < base + size - offset)
return addr - base;
pr_err("%s: unexpected addr! base 0x%llx size %zu offs %zu addr 0x%llx\n",
__func__, base, size, offset, addr);
WARN_ON(1);
return addr;
}
static phys_addr_t carveout_to_host_addr(union heap_options *options,
phys_addr_t addr)
{
phys_addr_t base = options->carveout.phys;
size_t size = options->carveout.size;
unsigned long offset = options->carveout.offs;
if (addr < size - offset)
return base + addr;
pr_err("%s: unexpected addr! base %llx size %zu offs %zu addr %#llx\n",
__func__, base, size, offset, addr);
WARN_ON(1);
return addr;
}
static void *carveout_get_kptr(phys_addr_t addr,
size_t size, enum img_mem_attr mattr)
{
/*
* Device memory is I/O memory and as a rule, it cannot
* be dereferenced safely without memory barriers, that
* is why it is guarded by __iomem (return of ioremap)
* and checked by sparse. It is accessed only through
* ioread32(), iowrit32(), etc.
*
* In x86 this memory can be dereferenced and safely
* accessed, i.e. a * __iomem pointer can be casted to
* a regular void* * pointer. We cast this here
* assuming FPGA is x86 and add __force to silence the
* sparse warning
*
* Note: System memory carveout can be used with cached turned on.
* */
void *kptr = NULL;
if (mattr & IMG_MEM_ATTR_UNCACHED)
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
kptr = (void * __force *)ioremap_nocache(addr, size);
#else
kptr = (void * __force *)ioremap(addr, size);
#endif
else if (mattr & IMG_MEM_ATTR_CACHED)
kptr = (void * __force *)ioremap_cache(addr, size);
else if (mattr & IMG_MEM_ATTR_WRITECOMBINE)
kptr = (void * __force *)ioremap_wc(addr, size);
return kptr;
}
static int carveout_put_kptr(void *addr)
{
iounmap((volatile void __iomem *)addr);
return 0;
}
#endif
static int vha_plat_probe(struct pci_dev *pci_dev,
const struct pci_device_id *id)
{
int bar, ret = 0;
struct imgpci_prvdata *data;
size_t maxmapsize = maxmapsizeMB * 1024 * 1024;
struct device *dev = &pci_dev->dev;
#ifdef CONFIG_GENERIC_ALLOCATOR
int heap;
#endif
dev_dbg(dev, "probing device, pci_dev: %p\n", dev);
/* Enable the device */
if (pci_enable_device(pci_dev))
goto out_free;
dev_info(dev, "%s dma_get_mask : %#llx\n", __func__, dma_get_mask(dev));
if (dev->dma_mask) {
dev_info(dev, "%s dev->dma_mask : %p : %#llx\n",
__func__, dev->dma_mask, *dev->dma_mask);
} else {
dev_info(dev, "%s mask unset, setting coherent\n", __func__);
dev->dma_mask = &dev->coherent_dma_mask;
}
dev_info(dev, "%s dma_set_mask %#llx\n", __func__, dma_get_mask(dev));
ret = dma_set_mask(dev, dma_get_mask(dev));
if (ret) {
dev_err(dev, "%s failed to set dma mask\n", __func__);
goto out_disable;
}
/* Reserve PCI I/O and memory resources */
if (pci_request_regions(pci_dev, "imgpci"))
goto out_disable;
/* Create a kernel space mapping for each of the bars */
data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
dev_dbg(dev, "allocated imgpci_prvdata @ %p\n", data);
memset(data, 0, sizeof(*data));
for (bar = 0; bar < NUM_EMU_BARS; bar++) {
data->memmap[bar].addr = pci_resource_start(pci_dev, bar);
data->memmap[bar].size = pci_resource_len(pci_dev, bar);
if (data->memmap[bar].size > maxmapsize) {
/*
* We avoid mapping too big regions: we do not need
* such a big amount of memory and some times we do
* not have enough contiguous 'vmallocable' memory.
*/
dev_warn(dev, "not mapping all mem for bar %u\n", bar);
data->memmap[bar].size = maxmapsize;
}
if (bar == NNA_MEM_BAR) {
/* Change memory size according to module parameter */
if (pci_size)
data->memmap[bar].size = pci_size;
/* ioremap fpga memory only when static mode is used */
if (!mem_static_kptr)
continue;
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
data->memmap[bar].km_addr = devm_ioremap_nocache(dev,
pci_resource_start(pci_dev, bar),
data->memmap[bar].size);
#else
data->memmap[bar].km_addr = devm_ioremap(dev,
pci_resource_start(pci_dev, bar),
data->memmap[bar].size);
#endif
dev_dbg(dev, "[bar %u] addr: 0x%lx size: 0x%lx km: 0x%p\n",
bar, data->memmap[bar].addr,
data->memmap[bar].size,
data->memmap[bar].km_addr);
}
/* Get the IRQ...*/
data->irq = pci_dev->irq;
data->pci_dev = pci_dev;
vha_pci_drv.pci_dev = pci_dev;
reset_emu(pci_dev, data);
if (!poll_interrupts) {
/* Enable interrupts */
emu_writereg32(data, PCI_EMU_SYS_CTRL_REGS_BAR,
PCI_EMU_INTERRUPT_ENABLE_OFS,
PCI_EMU_IRQ_ENABLE | PCI_EMU_IRQ_HIGH);
}
/* patch heap config with PCI memory addresses */
for (heap = 0; heap < vha_plat_emu_heaps; heap++) {
struct heap_config *cfg = &vha_plat_emu_heap_configs[heap];
#ifdef CONFIG_GENERIC_ALLOCATOR
if (cfg->type == IMG_MEM_HEAP_TYPE_CARVEOUT) {
cfg->options.carveout.phys = data->memmap[NNA_MEM_BAR].addr;
if (mem_static_kptr)
cfg->options.carveout.kptr =
data->memmap[NNA_MEM_BAR].km_addr;
cfg->options.carveout.size = data->memmap[NNA_MEM_BAR].size;
cfg->options.carveout.offs = pci_offset;
cfg->to_dev_addr = carveout_to_dev_addr;
cfg->to_host_addr = carveout_to_host_addr;
/* IO memory access callbacks */
if (!mem_static_kptr) {
/* Dynamic kernel memory mapping */
cfg->options.carveout.get_kptr = carveout_get_kptr;
cfg->options.carveout.put_kptr = carveout_put_kptr;
}
/* Allocation order */
cfg->options.carveout.pool_order = pool_alloc_order;
break;
}
#endif
}
ret = vha_add_dev(dev, vha_plat_emu_heap_configs,
vha_plat_emu_heaps, data,
data->memmap[NNA_REG_BAR].km_addr, data->memmap[NNA_REG_BAR].size);
if (ret) {
dev_err(dev, "failed to intialize driver core!\n");
goto out_release;
}
if (!poll_interrupts) {
/* Install the ISR callback...*/
ret = devm_request_threaded_irq(dev, data->irq, &pci_handle_irq,
&pci_thread_irq, IRQF_SHARED, DEVICE_NAME,
(void *)pci_dev);
if (ret) {
dev_err(dev, "failed to request irq!\n");
goto out_rm_dev;
}
dev_dbg(dev, "registered irq %d\n", data->irq);
} else {
INIT_DELAYED_WORK(&data->irq_work, pci_poll_interrupt);
data->irq_poll = 1;
/* Start the interrupt poll */
schedule_delayed_work(&data->irq_work,
usecs_to_jiffies(irq_poll_delay_us));
}
/* Try to calibrate the core if needed */
ret = vha_dev_calibrate(dev, FREQ_MEASURE_CYCLES);
if (ret) {
dev_err(dev, "%s: Failed to start clock calibration!\n", __func__);
goto out_rm_dev;
}
return ret;
out_rm_dev:
vha_rm_dev(dev);
out_release:
pci_release_regions(pci_dev);
out_disable:
pci_disable_device(pci_dev);
out_free:
return ret;
}
static void vha_plat_remove(struct pci_dev *dev)
{
dev_dbg(&dev->dev, "removing device\n");
pci_release_regions(dev);
pci_disable_device(dev);
vha_rm_dev(&dev->dev);
}
#ifdef CONFIG_PM
static int vha_plat_suspend(struct device *dev)
{
return vha_suspend_dev(dev);
}
static int vha_plat_resume(struct device *dev)
{
return vha_resume_dev(dev);
}
#endif
int vha_plat_init(void)
{
int ret;
ret = pci_register_driver(&vha_pci_drv.pci_driver);
if (ret) {
pr_err("failed to register PCI driver!\n");
return ret;
}
/* pci_dev should be set in probe */
if (!vha_pci_drv.pci_dev) {
pr_err("failed to find VHA PCI dev!\n");
pci_unregister_driver(&vha_pci_drv.pci_driver);
return -ENODEV;
}
return 0;
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

1004
drivers/nna/vha/platform/vha_plat_frost.c Normal file
View File

File diff suppressed because it is too large Load Diff

491
drivers/nna/vha/platform/vha_plat_nexef.c Normal file
View File

@@ -0,0 +1,491 @@
/*!
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/module.h>
#include <linux/device.h>
#include <linux/gfp.h>
#include <linux/dma-mapping.h>
#include <linux/pm.h>
#include <linux/mod_devicetable.h>
#include <linux/workqueue.h>
#include <linux/version.h>
#include <linux/of_platform.h>
#include "vha_common.h"
#include "vha_plat.h"
#include <nexef_plat.h>
/* NNPU TC exports we need*/
#include <tc_drv.h>
#define DEVICE_NAME "vha"
/*
* Special handling (not implemented) is required for the VHA device
* to be able to access both carveout buffers (internal memory) and
* dmabuf buffers (system memory).The latter have to go through
* the system bus to be accessed whereas the former do not.
*/
static struct heap_config vha_plat_fpga_heap_configs[] = {
/* Primary heap used for internal allocations */
#ifdef FPGA_BUS_MASTERING
#error Bus mastering not supported on this platform
#elif CONFIG_GENERIC_ALLOCATOR
{
.type = IMG_MEM_HEAP_TYPE_CARVEOUT,
/* .options.carveout to be filled at run time */
/* .to_dev_addr to be filled at run time */
},
#else
#error Neither FPGA_BUS_MASTERING or CONFIG_GENERIC_ALLOCATOR was defined
#endif
/* Secondary heap used for importing an external memory */
#ifdef CONFIG_DMA_SHARED_BUFFER
{
.type = IMG_MEM_HEAP_TYPE_DMABUF,
.to_dev_addr = NULL,
},
#else
#warning "Memory importing not supported!"
#endif
};
/* Number of core cycles used to measure the core clock frequency */
#define FREQ_MEASURE_CYCLES 0x7fffff
static const int vha_plat_fpga_heaps = sizeof(vha_plat_fpga_heap_configs)/
sizeof(*vha_plat_fpga_heap_configs);
static int vha_plat_probe(struct platform_device *pdev);
static int vha_plat_remove(struct platform_device *pdev);
static int vha_plat_suspend(struct platform_device *pdev, pm_message_t state);
static int vha_plat_resume(struct platform_device *pdev);
enum {
PLATFORM_IS_NEXEF = 1,
};
static struct platform_device_id nna_platform_device_id_table[] = {
{ .name = NEXEF_NNA_DEVICE_NAME, .driver_data = PLATFORM_IS_NEXEF },
{ },
};
static struct platform_driver vha_platform_drv = {
.probe = vha_plat_probe,
.remove = vha_plat_remove,
.suspend = vha_plat_suspend,
.resume = vha_plat_resume,
.driver = {
.owner = THIS_MODULE,
.name = DEVICE_NAME,
},
.id_table = nna_platform_device_id_table,
};
struct nna_driver_priv {
struct platform_device *pdev;
void __iomem *nna_regs;
uint32_t nna_size;
/* Work for the threaded interrupt. */
struct work_struct work;
};
/*
* reset_dut - Reset the Device Under Test
*/
static void reset_dut(struct device *dev)
{
/* Nothing yet until Odin baseboard is updated to support that */
//tc_dut2_reset(dev);
}
/*
* pci_thread_irq - High latency interrupt handler
*/
static void nna_soft_isr_cb(struct work_struct *work)
{
struct nna_driver_priv *priv = container_of(work, struct nna_driver_priv, work);
struct platform_device *pdev = priv->pdev;
struct device *dev = &pdev->dev;
vha_handle_thread_irq(dev);
}
/*
* pci_isr_cb - Low latency interrupt handler
*/
static void nna_hard_isr_cb(void *pdev_id)
{
struct platform_device *pdev = (struct platform_device *)pdev_id;
struct device *dev = &pdev->dev;
struct nna_driver_priv *priv = (struct nna_driver_priv *)vha_get_plat_data(dev);
irqreturn_t ret = IRQ_NONE;
ret = vha_handle_irq(dev);
if (ret == IRQ_WAKE_THREAD) {
schedule_work(&priv->work);
}
}
#ifdef CONFIG_GENERIC_ALLOCATOR
static phys_addr_t carveout_to_dev_addr(union heap_options *options,
phys_addr_t addr)
{
phys_addr_t base = options->carveout.phys;
size_t size = options->carveout.size;
unsigned long offset = options->carveout.offs;
if (addr - offset >= base && addr < base + size - offset)
return addr - base;
pr_err("%s: unexpected addr! base 0x%llx size %zu offs %zu addr 0x%llx\n",
__func__, base, size, offset, addr);
WARN_ON(1);
return addr;
}
static phys_addr_t carveout_to_host_addr(union heap_options *options,
phys_addr_t addr)
{
phys_addr_t base = options->carveout.phys;
size_t size = options->carveout.size;
unsigned long offset = options->carveout.offs;
if (addr < size - offset)
return base + addr;
pr_err("%s: unexpected addr! base %llx size %zu offs %zu addr %#llx\n",
__func__, base, size, offset, addr);
WARN_ON(1);
return addr;
}
static void *carveout_get_kptr(phys_addr_t addr,
size_t size, enum img_mem_attr mattr)
{
/*
* Device memory is I/O memory and as a rule, it cannot
* be dereferenced safely without memory barriers, that
* is why it is guarded by __iomem (return of ioremap)
* and checked by sparse. It is accessed only through
* ioread32(), iowrit32(), etc.
*
* In x86 this memory can be dereferenced and safely
* accessed, i.e. a * __iomem pointer can be casted to
* a regular void* * pointer. We cast this here
* assuming FPGA is x86 and add __force to silence the
* sparse warning
*
* Note: System memory carveout can be used with cached turned on.
* */
void *kptr = NULL;
if (mattr & IMG_MEM_ATTR_UNCACHED)
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
kptr = (void * __force *)ioremap_nocache(addr, size);
#else
kptr = (void * __force *)ioremap(addr, size);
#endif
else if (mattr & IMG_MEM_ATTR_CACHED)
kptr = (void * __force *)ioremap_cache(addr, size);
else if (mattr & IMG_MEM_ATTR_WRITECOMBINE)
kptr = (void * __force *)ioremap_wc(addr, size);
/*pr_debug(
"Mapping %zu bytes into kernel memory (Phys:%08llX, Kptr:%p)\n",
size, addr, kptr);
pr_debug("[%c%c%c]\n",
(mattr & IMG_MEM_ATTR_UNCACHED) ? 'U' : '.',
(mattr & IMG_MEM_ATTR_CACHED) ? 'C' : '.',
(mattr & IMG_MEM_ATTR_WRITECOMBINE) ? 'W' : '.');*/
return kptr;
}
static int carveout_put_kptr(void *addr)
{
/* pr_debug("Unmapping kernel memory (Phys: %p)\n", addr);*/
iounmap(addr);
return 0;
}
#endif
static int vha_plat_probe(struct platform_device *pdev)
{
int ret = 0;
struct nna_driver_priv *priv;
struct device *dev = &pdev->dev;
struct nexef_nna_platform_data *platdata;
struct resource *nna_registers;
uint64_t vha_mem_base, vha_mem_size;
uint64_t vha_mem_phys_offset = 0;
int heap;
dev_info(dev, "%s dma_get_mask : %#llx\n", __func__, dma_get_mask(dev));
priv = devm_kzalloc(dev, sizeof(struct nna_driver_priv), GFP_KERNEL);
if (!priv) {
ret = -ENOMEM;
goto out_no_free;
}
priv->pdev = pdev;
if (dev->dma_mask) {
dev_info(dev, "%s dev->dma_mask : %p : %#llx\n",
__func__, dev->dma_mask, *dev->dma_mask);
} else {
dev_info(dev, "%s mask unset, setting coherent\n", __func__);
dev->dma_mask = &dev->coherent_dma_mask;
}
dev_info(dev, "%s dma_set_mask %#llx\n", __func__, dma_get_mask(dev));
ret = dma_set_mask(dev, dma_get_mask(dev));
if (ret) {
dev_err(dev, "%s failed to set dma mask\n", __func__);
goto out_dma_free;
}
/* Allocate dut2 registers */
nna_registers = platform_get_resource_byname(pdev,
IORESOURCE_MEM, "nna-regs");
if (!nna_registers) {
ret = -EIO;
goto out_dma_free;
}
priv->nna_regs = devm_ioremap_resource(dev, nna_registers);
if (!priv->nna_regs) {
ret = -EIO;
goto out_dma_free;
}
priv->nna_size = nna_registers->end - nna_registers->start;
/* Get infos for DUT memory */
platdata = dev_get_platdata(dev);
/* Get out mem specs */
vha_mem_size = platdata->nna_memory_size;
vha_mem_base = platdata->nna_memory_base;
vha_mem_phys_offset = platdata->nna_memory_offset;
dev_dbg(dev, "PCI memory: base: %#llX - size: %#llX - offset: %#llX",
vha_mem_base, vha_mem_size, vha_mem_phys_offset);
/* patch heap config with PCI memory addresses */
for (heap = 0; heap < vha_plat_fpga_heaps; heap++) {
struct heap_config *cfg = &vha_plat_fpga_heap_configs[heap];
switch(cfg->type) {
case IMG_MEM_HEAP_TYPE_CARVEOUT:
cfg->options.carveout.phys = vha_mem_base;
cfg->options.carveout.size = vha_mem_size;
cfg->options.carveout.offs = vha_mem_phys_offset;
cfg->to_dev_addr = carveout_to_dev_addr;
cfg->to_host_addr = carveout_to_host_addr;
/* IO memory access callbacks */
cfg->options.carveout.get_kptr = carveout_get_kptr;
cfg->options.carveout.put_kptr = carveout_put_kptr;
break;
case IMG_MEM_HEAP_TYPE_DMABUF: /* Nothing to do here */
break;
default:
dev_err(dev, "Unsupported heap type %d!\n", cfg->type);
break;
}
}
reset_dut(dev->parent);
ret = vha_add_dev(dev,
vha_plat_fpga_heap_configs,
vha_plat_fpga_heaps,
priv,
priv->nna_regs,
priv->nna_size);
if (ret) {
dev_err(dev, "failed to initialize driver core!\n");
goto out_dma_free;
}
/*
* Reset FPGA DUT only after disabling clocks in
* vha_add_dev()-> get properties.
* This workaround is required to ensure that
* clocks (on daughter board) are enabled for test slave scripts to
* read FPGA build version register.
*/
reset_dut(dev->parent);
/* Install the ISR callback...*/
INIT_WORK(&priv->work, nna_soft_isr_cb);
ret = tc_set_interrupt_handler(dev->parent, TC_INTERRUPT_NNA, nna_hard_isr_cb, pdev);
ret |= tc_enable_interrupt(dev->parent, TC_INTERRUPT_NNA);
if (ret) {
dev_err(dev, "failed to request irq!\n");
goto out_rm_dev;
}
/* Try to calibrate the core if needed */
ret = vha_dev_calibrate(dev, FREQ_MEASURE_CYCLES);
if (ret) {
dev_err(dev, "%s: Failed to start clock calibration!\n", __func__);
goto out_rm_dev;
}
return ret;
out_rm_dev:
/* Disable interrupt handler just in case it is enable which fail */
tc_set_interrupt_handler(dev->parent, TC_INTERRUPT_NNA, NULL, NULL);
vha_rm_dev(dev);
out_dma_free:
#if LINUX_VERSION_CODE < KERNEL_VERSION(5,4,0)
/* Release any declared mem regions */
dma_release_declared_memory(dev);
#endif
out_no_free:
return ret;
}
static int vha_plat_remove(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct nna_driver_priv *priv =
(struct nna_driver_priv *)vha_get_plat_data(dev);
dev_dbg(dev, "removing device\n");
/* Disable interrupts */
tc_disable_interrupt(dev->parent, TC_INTERRUPT_NNA);
tc_set_interrupt_handler(dev->parent, TC_INTERRUPT_NNA, NULL, NULL);
/* Make sure there is no work in the queue */
if (priv)
cancel_work_sync(&priv->work);
#if LINUX_VERSION_CODE < KERNEL_VERSION(5,4,0)
/* Release any declared mem regions */
dma_release_declared_memory(dev);
#endif
vha_rm_dev(dev);
return 0;
}
#ifdef CONFIG_PM
static int vha_plat_suspend(struct platform_device *pdev, pm_message_t state)
{
struct device *dev = &pdev->dev;
return vha_suspend_dev(dev);
}
static int vha_plat_resume(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
return vha_resume_dev(dev);
}
#endif
/* Functions called by vha_core */
int vha_plat_init(void)
{
int ret;
ret = platform_driver_register(&vha_platform_drv);
if (ret) {
pr_err("failed to register platform driver!\n");
return ret;
}
return 0;
}
int vha_plat_deinit(void)
{
int ret;
//reset_dut();
platform_driver_unregister(&vha_platform_drv);
ret = vha_deinit();
if (ret)
pr_err("VHA driver deinit failed\n");
return ret;
}
/*
* NOTE: customer may want to use spinlock to avoid
* problems with multi threaded IO access.
*
*/
uint64_t vha_plat_read64(void *addr)
{
return (uint64_t)readl(addr) | ((uint64_t)readl(addr + 4) << 32);
}
void vha_plat_write64(void *addr, uint64_t val)
{
writel(val & 0xffffffff, addr);
writel(((uint64_t)val >> 32), addr + 4);
}

1152
drivers/nna/vha/platform/vha_plat_odin.c Normal file
View File

File diff suppressed because it is too large Load Diff

1065
drivers/nna/vha/platform/vha_plat_orion.c Normal file
View File

File diff suppressed because it is too large Load Diff

36
drivers/nna/vha/platform/vha_plat_param_thead_light_fpga_c910.h Normal file
View File

@@ -0,0 +1,36 @@
/*!
*****************************************************************************
*
* @File vha_plat_param_thead_light_fpga_c910.h
* ---------------------------------------------------------------------------
*
* Copyright (C) 2020 Alibaba Group Holding Limited
*
*****************************************************************************/
#ifndef VHA_PLAT_PARAM_THEAD_LIGHT_FPGA_C910_H
#define VHA_PLAT_PARAM_THEAD_LIGHT_FPGA_C910_H
/* Core clock frequency: default 30MHz */
#define VHA_CORE_CLOCK_MHZ 1000
/* Core watchdog cycles default value */
/* MMM can transfer any number of bytes at cost of higher cycles, setting it to ~100ms @800MHz */
#define VHA_CORE_WDT_CYCLES (0x4ffffff*VHA_CORE_CLOCK_MHZ/800)
/* Memory watchdog is set ~1ms @800MHz which is very safe value to avoid any false interrupts */
#define VHA_MEM_WDT_CYCLES (0xfffff*VHA_CORE_CLOCK_MHZ/800)
/* Memory burst size */
#define VHA_CORE_MH_MAX_BURST_LENGTH 128
/* SLC cache policy type (0-use cache, 1-bypass cache) */
#define VHA_CORE_MH_SLC_CACHE_POLICY_TYPE 1
/* GPU pipe coherent type */
#define VHA_CORE_MH_GPU_PIPE_COHERENT_TYPE 1
/* Persistence priority 0-lowest,3-highest */
#define VHA_CORE_MH_PERSISTENCE_PRIO 0
/* Suspend delay in ms after which the
* runtime suspend callback is called */
#define VHA_CORE_SUSPEND_DELAY 10
#endif /* VHA_PLAT_PARAM_THEAD_LIGHT_FPGA_C910_H */

483
drivers/nna/vha/platform/vha_plat_pci.c Normal file
View File

@@ -0,0 +1,483 @@
/*!
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/module.h>
#include <linux/device.h>
#include <linux/gfp.h>
#include <linux/dma-mapping.h>
#include <linux/pci.h>
#include <linux/pm.h>
#include <linux/mod_devicetable.h>
#include "uapi/version.h"
#include "vha_common.h"
#include "vha_plat.h"
#define DEVICE_NAME "vha"
static unsigned long carveout_phys_start;
module_param(carveout_phys_start, ulong, 0444);
MODULE_PARM_DESC(carveout_phys_start,
"Physical address of start of carveout region");
static uint32_t carveout_size;
module_param(carveout_size, uint, 0444);
MODULE_PARM_DESC(carveout_size,
"Size of carveout region: takes precedence over any PCI based memory");
#define IMG_PCI_VENDOR_ID 0x1010
#define IMG_PCI_DEVICE_ID 0x1002
#define PCI_BAR_DEV 0
/* PCI and PCI-E do not support 64bit devices on BAR1: 64bit uses 2 bars */
#define PCI_BAR_RAM 2
#define NUM_PCI_BARS 3
/* Number of core cycles used to measure the core clock frequency */
#define FREQ_MEASURE_CYCLES 0x7fffff
static struct heap_config vha_plat_pci_heap_configs[] = {
/* first entry is the default heap */
{
.type = IMG_MEM_HEAP_TYPE_CARVEOUT,
/* .options.carveout to be filled at run time */
/* .to_dev_addr to be filled at run time */
/* .to_host_addr to be filled at run time */
},
{
.type = IMG_MEM_HEAP_TYPE_DMABUF,
/*.to_dev_addr = NULL,*/
/*.to_host_addr = NULL,*/
},
};
static const int vha_plat_fpga_heaps =
sizeof(vha_plat_pci_heap_configs)/sizeof(*vha_plat_pci_heap_configs);
static const struct pci_device_id pci_pci_ids[] = {
{ PCI_DEVICE(IMG_PCI_VENDOR_ID, IMG_PCI_DEVICE_ID), },
{ 0, }
};
MODULE_DEVICE_TABLE(pci, pci_pci_ids);
struct imgpci_prvdata {
int irq;
struct {
unsigned long addr;
unsigned long size;
void __iomem *km_addr;
} memmap[NUM_PCI_BARS];
struct pci_dev *pci_dev;
};
struct img_pci_driver {
struct pci_dev *pci_dev;
struct pci_driver pci_driver;
};
static int vha_plat_probe(struct pci_dev *pci_dev,
const struct pci_device_id *id);
static void vha_plat_remove(struct pci_dev *dev);
static int vha_plat_suspend(struct device *dev);
static int vha_plat_resume(struct device *dev);
static SIMPLE_DEV_PM_OPS(vha_pm_plat_ops,
vha_plat_suspend, vha_plat_resume);
static ssize_t info_show(struct device_driver *drv, char *buf)
{
return sprintf(buf, "VHA dummy driver version : " VERSION_STRING "\n");
}
static DRIVER_ATTR_RO(info);
static struct attribute *drv_attrs[] = {
&driver_attr_info.attr,
NULL
};
ATTRIBUTE_GROUPS(drv);
static struct img_pci_driver vha_pci_drv = {
.pci_driver = {
.name = "vha_pci",
.id_table = pci_pci_ids,
.probe = vha_plat_probe,
.remove = vha_plat_remove,
.driver = {
.groups = drv_groups,
.pm = &vha_pm_plat_ops,
}
},
};
static ulong maxmapsizeMB = (sizeof(void *) == 4) ? 400 : 1024;
static irqreturn_t pci_thread_irq(int irq, void *dev_id)
{
struct pci_dev *dev = (struct pci_dev *)dev_id;
return vha_handle_thread_irq(&dev->dev);
}
static irqreturn_t pci_handle_irq(int irq, void *dev_id)
{
struct pci_dev *dev = (struct pci_dev *)dev_id;
irqreturn_t ret = IRQ_NONE;
ret = vha_handle_irq(&dev->dev);
return ret;
}
/*
* IO hooks.
* NOTE: customer may want to use spinlock to avoid
* problems with multi threaded IO access
*/
uint64_t vha_plat_read64(void *addr)
{
return readq(addr);
}
void vha_plat_write64(void *addr, uint64_t val)
{
writeq(val, addr);
}
int vha_plat_deinit(void)
{
int ret;
/* Unregister the driver from the OS */
pci_unregister_driver(&(vha_pci_drv.pci_driver));
ret = vha_deinit();
if (ret)
pr_err("VHA driver deinit failed\n");
return ret;
}
#ifdef CONFIG_GENERIC_ALLOCATOR
static phys_addr_t carveout_to_dev_addr(union heap_options *options,
phys_addr_t addr)
{
phys_addr_t base = options->carveout.phys;
size_t size = options->carveout.size;
if (addr >= base && addr < base + size)
return addr - base;
pr_err("%s: unexpected addr! base %llx size %zu addr %#llx\n",
__func__, base, size, addr);
WARN_ON(1);
return addr;
}
static phys_addr_t carveout_to_host_addr(union heap_options *options,
phys_addr_t addr)
{
phys_addr_t base = options->carveout.phys;
size_t size = options->carveout.size;
if (addr < size)
return base + addr;
pr_err("%s: unexpected addr! base %llx size %zu addr %#llx\n",
__func__, base, size, addr);
WARN_ON(1);
return addr;
}
static void *carveout_get_kptr(phys_addr_t addr,
size_t size, enum img_mem_attr mattr)
{
/*
* Device memory is I/O memory and as a rule, it cannot
* be dereferenced safely without memory barriers, that
* is why it is guarded by __iomem (return of ioremap)
* and checked by sparse. It is accessed only through
* ioread32(), iowrit32(), etc.
*
* In x86 this memory can be dereferenced and safely
* accessed, i.e. a * __iomem pointer can be casted to
* a regular void* * pointer. We cast this here
* assuming FPGA is x86 and add __force to silence the
* sparse warning
*
* Note: System memory carveout can be used with cached turned on.
* */
void *kptr = NULL;
if (mattr & IMG_MEM_ATTR_UNCACHED)
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
kptr = (void * __force *)ioremap_nocache(addr, size);
#else
kptr = (void * __force *)ioremap(addr, size);
#endif
else if (mattr & IMG_MEM_ATTR_CACHED)
kptr = (void * __force *)ioremap_cache(addr, size);
else if (mattr & IMG_MEM_ATTR_WRITECOMBINE)
kptr = (void * __force *)ioremap_wc(addr, size);
return kptr;
}
static int carveout_put_kptr(void *addr)
{
iounmap(addr);
return 0;
}
#endif
static int vha_plat_probe(struct pci_dev *pci_dev,
const struct pci_device_id *id)
{
int bar, ret = 0;
struct imgpci_prvdata *data;
size_t maxmapsize = maxmapsizeMB * 1024 * 1024;
struct device *dev = &pci_dev->dev;
#ifdef CONFIG_GENERIC_ALLOCATOR
int heap;
#endif
dev_info(dev, "probed a VHA device, pci_dev: %x:%x\n",
pci_dev->vendor, pci_dev->device);
/* Enable the device */
if (pci_enable_device(pci_dev))
goto out_free;
if (dev->dma_mask) {
dev_info(dev, "%s dev->dma_mask : %#llx\n",
__func__, *dev->dma_mask);
} else {
dev_info(dev, "%s mask unset, setting coherent\n", __func__);
dev->dma_mask = &dev->coherent_dma_mask;
}
dev_info(dev, "%s dma_set_mask %#llx\n", __func__, dma_get_mask(dev));
ret = dma_set_mask(dev, dma_get_mask(dev));
if (ret) {
dev_err(dev, "%s failed to set dma mask\n", __func__);
goto out_disable;
}
/* Reserve PCI I/O and memory resources */
if (pci_request_regions(pci_dev, "imgpci"))
goto out_disable;
/* Create a kernel space mapping for each of the bars */
data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
dev_dbg(dev, "allocated imgpci_prvdata @ %p\n", data);
memset(data, 0, sizeof(*data));
for (bar = 0; bar < NUM_PCI_BARS; bar += PCI_BAR_RAM-PCI_BAR_DEV) {
data->memmap[bar].addr = pci_resource_start(pci_dev, bar);
data->memmap[bar].size = pci_resource_len(pci_dev, bar);
if (bar == PCI_BAR_RAM) {
/* Don't ioremap pci memory - it is mapped on demand */
continue;
}
if (data->memmap[bar].size > maxmapsize) {
/*
* We avoid mapping too big regions: we do not need
* such a big amount of memory and some times we do
* not have enough contiguous 'vmallocable' memory.
*/
dev_warn(dev, "not mapping all mem for bar %u\n", bar);
data->memmap[bar].size = maxmapsize;
}
data->memmap[bar].km_addr = devm_ioremap(dev,
pci_resource_start(pci_dev, bar),
data->memmap[bar].size);
dev_info(dev, "[bar %u] addr: 0x%lx size: 0x%lx km: 0x%p\n",
bar, data->memmap[bar].addr,
data->memmap[bar].size,
data->memmap[bar].km_addr);
}
/* Get the IRQ...*/
data->irq = pci_dev->irq;
data->pci_dev = pci_dev;
vha_pci_drv.pci_dev = pci_dev;
/* patch heap config with PCI memory addresses */
for (heap = 0; heap < vha_plat_fpga_heaps; heap++) {
struct heap_config *cfg = &vha_plat_pci_heap_configs[heap];
#ifdef CONFIG_GENERIC_ALLOCATOR
if (cfg->type == IMG_MEM_HEAP_TYPE_CARVEOUT) {
if (carveout_size && carveout_phys_start) {
/*
* 2 types of carveout memory are supported:
* memory carved out of the main DDR
* memory region.
* eg: linux boot option memmap=512M$0x5CAFFFFF
* This is configured using module parameters:
* contig_phys_start and size
* DDR populated in the actual PCI card,
* in BAR 4.
* The module parameters take precedence
* over PCI memory.
*/
cfg->options.carveout.phys =
carveout_phys_start;
cfg->options.carveout.size =
carveout_size;
cfg->to_dev_addr = carveout_to_dev_addr;
cfg->to_host_addr = carveout_to_host_addr;
dev_info(dev, "using %dMB CARVEOUT at x%lx\n",
carveout_size/1024/1024,
carveout_phys_start);
} else if (data->memmap[PCI_BAR_RAM].size) {
cfg->options.carveout.phys =
data->memmap[PCI_BAR_RAM].addr;
if (carveout_size)
cfg->options.carveout.size =
carveout_size;
else
cfg->options.carveout.size =
data->memmap[PCI_BAR_RAM].size;
cfg->to_dev_addr = carveout_to_dev_addr;
cfg->to_host_addr = carveout_to_host_addr;
dev_info(dev, "using %ldMB CARVEOUT from PCI bar %d\n",
cfg->options.carveout.size/1024/1024,
PCI_BAR_RAM);
}
/* IO memory access callbacks */
cfg->options.carveout.get_kptr = carveout_get_kptr;
cfg->options.carveout.put_kptr = carveout_put_kptr;
break;
}
#endif
/* THIS IS HACKY - just for testing dmabuf importing on qemu.
* Dma buf config - using carveout type for dmabuf exporter.
* Assuming memory just after carveout_phys_start +
* carveout_size and size of carveout_size */
if (cfg->type == IMG_MEM_HEAP_TYPE_DMABUF) {
cfg->options.carveout.phys =
carveout_phys_start + carveout_size;
cfg->options.carveout.size = carveout_size;
cfg->to_dev_addr = carveout_to_dev_addr;
}
}
ret = vha_add_dev(dev, vha_plat_pci_heap_configs,
vha_plat_fpga_heaps, data,
data->memmap[PCI_BAR_DEV].km_addr,
data->memmap[PCI_BAR_DEV].size);
if (ret) {
dev_err(dev, "failed to intialize driver core!\n");
goto out_release;
}
/* Install the ISR callback...*/
ret = devm_request_threaded_irq(dev, data->irq, &pci_handle_irq,
&pci_thread_irq, IRQF_SHARED, DEVICE_NAME,
(void *)pci_dev);
if (ret) {
dev_err(dev, "failed to request irq!\n");
goto out_rm_dev;
}
dev_dbg(dev, "registerd irq %d\n", data->irq);
/* Try to calibrate the core if needed */
ret = vha_dev_calibrate(dev, FREQ_MEASURE_CYCLES);
if (ret) {
dev_err(dev, "%s: Failed to start clock calibration!\n", __func__);
goto out_rm_dev;
}
return ret;
out_rm_dev:
vha_rm_dev(dev);
out_release:
pci_release_regions(pci_dev);
out_disable:
pci_disable_device(pci_dev);
out_free:
return ret;
}
static void vha_plat_remove(struct pci_dev *dev)
{
dev_dbg(&dev->dev, "removing device\n");
pci_release_regions(dev);
pci_disable_device(dev);
vha_rm_dev(&dev->dev);
}
#ifdef CONFIG_PM
static int vha_plat_suspend(struct device *dev)
{
return vha_suspend_dev(dev);
}
static int vha_plat_resume(struct device *dev)
{
return vha_resume_dev(dev);
}
#endif
int vha_plat_init(void)
{
int ret;
ret = pci_register_driver(&vha_pci_drv.pci_driver);
if (ret) {
pr_err("failed to register PCI driver!\n");
return ret;
}
/* pci_dev should be set in probe */
if (!vha_pci_drv.pci_dev) {
pr_err("failed to find VHA PCI dev!\n");
pci_unregister_driver(&vha_pci_drv.pci_driver);
return -ENODEV;
}
return 0;
}

371
drivers/nna/vha/platform/vha_plat_thead.c Normal file
View File

@@ -0,0 +1,371 @@
/*!
*****************************************************************************
*
* @File vha_plat_thead.c
* ---------------------------------------------------------------------------
*
* Copyright (C) 2020 Alibaba Group Holding Limited
*
*****************************************************************************/
#define DEBUG
#include <linux/interrupt.h>
#include <linux/device.h>
#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/io.h>
#include <linux/pm.h>
#include <linux/version.h>
#include <img_mem_man.h>
#include "vha_common.h"
#include "uapi/version.h"
#include "vha_plat.h"
#include "vha_plat_dt.h"
#if defined(CFG_SYS_VAGUS)
#include <hwdefs/vagus_system.h>
#elif defined(CFG_SYS_AURA)
#include <hwdefs/aura_system.h>
#elif defined(CFG_SYS_MIRAGE)
#include <hwdefs/mirage_system.h>
#endif
#define DEVICE_NAME "vha"
/* Number of core cycles used to measure the core clock frequency */
#define FREQ_MEASURE_CYCLES 0xfffffff
static bool poll_interrupts; /* Disabled by default */
module_param(poll_interrupts, bool, 0444);
MODULE_PARM_DESC(poll_interrupts, "Poll for interrupts? 0: No, 1: Yes");
static unsigned int irq_poll_interval_ms = 100; /* 100 ms */
module_param(irq_poll_interval_ms, uint, 0444);
MODULE_PARM_DESC(irq_poll_interval_ms, "Time in ms between each interrupt poll");
/* Global timer used when irq poll mode is switched on.
* NOTE: only single core instance is supported in polling mode */
static struct poll_timer {
struct platform_device *pdev;
struct timer_list tmr;
bool enabled;
} irq_poll_timer;
static ssize_t info_show(struct device_driver *drv, char *buf);
static irqreturn_t dt_plat_thread_irq(int irq, void *dev_id)
{
struct platform_device *ofdev = (struct platform_device *)dev_id;
return vha_handle_thread_irq(&ofdev->dev);
}
static irqreturn_t dt_plat_isrcb(int irq, void *dev_id)
{
struct platform_device *ofdev = (struct platform_device *)dev_id;
if (!ofdev)
return IRQ_NONE;
return vha_handle_irq(&ofdev->dev);
}
/* Interrupt polling function */
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,15,0)
static void dt_plat_poll_interrupt(struct timer_list *t)
{
struct poll_timer *poll_timer = from_timer(poll_timer, t, tmr);
#else
static void dt_plat_poll_interrupt(unsigned long ctx)
{
struct poll_timer *poll_timer = (struct poll_timer *)ctx;
#endif
struct platform_device *ofdev = poll_timer->pdev;
int ret;
if (!poll_timer->enabled)
return;
preempt_disable();
ret = vha_handle_irq(&ofdev->dev);
preempt_enable();
if (ret == IRQ_WAKE_THREAD)
vha_handle_thread_irq(&ofdev->dev);
/* retrigger */
mod_timer(&poll_timer->tmr,
jiffies + msecs_to_jiffies(irq_poll_interval_ms));
}
static int vha_plat_probe(struct platform_device *ofdev)
{
int ret, module_irq;
struct resource res;
void __iomem *reg_addr;
uint32_t reg_size, core_size;
char info[256];
info_show(ofdev->dev.driver, info);
pr_info("%s: Version: %s\n", __func__, info);
ret = of_address_to_resource(ofdev->dev.of_node, 0, &res);
if (ret) {
dev_err(&ofdev->dev, "missing 'reg' property in device tree\n");
return ret;
}
pr_info("%s: registers %#llx-%#llx\n", __func__,
(unsigned long long)res.start, (unsigned long long)res.end);
module_irq = irq_of_parse_and_map(ofdev->dev.of_node, 0);
if (module_irq == 0) {
dev_err(&ofdev->dev, "could not map IRQ\n");
return -ENXIO;
}
/* Assuming DT holds a single registers space entry that covers all regions,
* So we need to do the split accordingly */
reg_size = res.end - res.start + 1;
#ifdef CFG_SYS_VAGUS
core_size = _REG_SIZE + _REG_NNSYS_SIZE;
#else
core_size = _REG_SIZE;
#endif
if ((res.start + _REG_START) > res.end) {
dev_err(&ofdev->dev, "wrong system conf for core region!\n");
return -ENXIO;
}
if ((res.start + _REG_START + core_size) > res.end) {
dev_warn(&ofdev->dev, "trimming system conf for core region!\n");
core_size = reg_size - _REG_START;
}
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 6, 0)
reg_addr = devm_ioremap_nocache(&ofdev->dev, res.start +
_REG_START, core_size);
#else
reg_addr = devm_ioremap(&ofdev->dev, res.start +
_REG_START, core_size);
#endif
if (!reg_addr) {
dev_err(&ofdev->dev, "failed to map core registers\n");
return -ENXIO;
}
ret = vha_plat_dt_hw_init(ofdev);
if (ret) {
dev_err(&ofdev->dev, "failed to init platform-specific hw!\n");
goto out_add_dev;
}
/* no 'per device' memory heaps used */
ret = vha_add_dev(&ofdev->dev, NULL, 0,
NULL /* plat priv data */, reg_addr, core_size);
if (ret) {
dev_err(&ofdev->dev, "failed to intialize driver core!\n");
goto out_add_dev;
}
if (!poll_interrupts) {
ret = devm_request_threaded_irq(&ofdev->dev, module_irq, &dt_plat_isrcb,
&dt_plat_thread_irq, IRQF_SHARED, DEVICE_NAME, ofdev);
if (ret) {
dev_err(&ofdev->dev, "failed to request irq\n");
goto out_irq;
}
} else {
irq_poll_timer.pdev = ofdev;
irq_poll_timer.enabled = true;
/* Setup and start poll timer */
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4,15,0)
timer_setup(&irq_poll_timer.tmr, dt_plat_poll_interrupt, 0);
#else
setup_timer(&irq_poll_timer.tmr, dt_plat_poll_interrupt,
(uintptr_t)&irq_poll_timer);
#endif
mod_timer(&irq_poll_timer.tmr,
jiffies + msecs_to_jiffies(irq_poll_interval_ms));
}
/* Try to calibrate the core if needed */
ret = vha_dev_calibrate(&ofdev->dev, FREQ_MEASURE_CYCLES);
if (ret) {
dev_err(&ofdev->dev, "%s: Failed to start clock calibration!\n", __func__);
goto out_irq;
}
return ret;
out_irq:
vha_rm_dev(&ofdev->dev);
out_add_dev:
devm_iounmap(&ofdev->dev, reg_addr);
return ret;
}
static int vha_plat_remove(struct platform_device *ofdev)
{
vha_rm_dev(&ofdev->dev);
vha_plat_dt_hw_destroy(ofdev);
return 0;
}
#ifdef CONFIG_PM
static int vha_plat_suspend(struct device *dev)
{
struct platform_device *ofdev =
container_of(dev, struct platform_device, dev);
int ret = 0;
ret = vha_suspend_dev(dev);
if (ret)
dev_err(dev, "failed to suspend the core!\n");
else {
ret = vha_plat_dt_hw_suspend(ofdev);
if (ret)
dev_err(dev, "failed to suspend platform-specific hw!\n");
}
return ret;
}
static int vha_plat_resume(struct device *dev)
{
struct platform_device *ofdev =
container_of(dev, struct platform_device, dev);
int ret = 0;
ret = vha_plat_dt_hw_resume(ofdev);
if (ret)
dev_err(dev, "failed to resume platform-specific hw!\n");
else {
ret = vha_resume_dev(dev);
if (ret)
dev_err(dev, "failed to resume the core!\n");
}
return ret;
}
static int vha_plat_runtime_idle(struct device *dev)
{
return 0;
}
static int vha_plat_runtime_suspend(struct device *dev)
{
struct platform_device *ofdev =
container_of(dev, struct platform_device, dev);
int ret = 0;
ret = vha_plat_dt_hw_suspend(ofdev);
if (ret)
dev_err(dev, "failed to suspend platform-specific hw!\n");
return ret;
}
static int vha_plat_runtime_resume(struct device *dev)
{
struct platform_device *ofdev =
container_of(dev, struct platform_device, dev);
int ret = 0;
ret = vha_plat_dt_hw_resume(ofdev);
if (ret)
dev_err(dev, "failed to resume platform-specific hw!\n");
return ret;
}
#endif
static struct dev_pm_ops vha_pm_plat_ops = {
SET_RUNTIME_PM_OPS(vha_plat_runtime_suspend,
vha_plat_runtime_resume, NULL)
SET_SYSTEM_SLEEP_PM_OPS(vha_plat_suspend, vha_plat_resume)
};
static ssize_t info_show(struct device_driver *drv, char *buf)
{
return sprintf(buf, "VHA DT driver version : " VERSION_STRING "\n");
}
static DRIVER_ATTR_RO(info);
static struct attribute *drv_attrs[] = {
&driver_attr_info.attr,
NULL
};
ATTRIBUTE_GROUPS(drv);
static struct platform_driver vha_plat_drv = {
.probe = vha_plat_probe,
.remove = vha_plat_remove,
.driver = {
.name = "ax3xxx-nna",
.groups = drv_groups,
.owner = THIS_MODULE,
.of_match_table = vha_plat_dt_of_ids,
.pm = &vha_pm_plat_ops,
},
};
int vha_plat_init(void)
{
int ret = 0;
struct heap_config *heap_configs;
int num_heaps;
vha_plat_dt_get_heaps(&heap_configs, &num_heaps);
ret = vha_init_plat_heaps(heap_configs, num_heaps);
if (ret) {
pr_err("failed to initialize global heaps\n");
return -ENOMEM;
}
ret = platform_driver_register(&vha_plat_drv);
if (ret) {
pr_err("failed to register VHA driver!\n");
return ret;
}
return 0;
}
int vha_plat_deinit(void)
{
int ret;
if (poll_interrupts) {
irq_poll_timer.enabled = false;
del_timer_sync(&irq_poll_timer.tmr);
}
/* Unregister the driver from the OS */
platform_driver_unregister(&vha_plat_drv);
ret = vha_deinit();
if (ret)
pr_err("VHA driver deinit failed\n");
return ret;
}
/*
* coding style for emacs
*
* Local variables:
* indent-tabs-mode: t
* tab-width: 8
* c-basic-offset: 8
* End:
*/

181
drivers/nna/vha/platform/vha_plat_thead_light.c Normal file
View File

@@ -0,0 +1,181 @@
/*!
*****************************************************************************
*
* @File vha_plat_thead_light_fpga_c910.c
* ---------------------------------------------------------------------------
*
* Copyright (C) 2020 Alibaba Group Holding Limited
*
*****************************************************************************/
#include <linux/module.h>
#include <linux/gfp.h>
#include <linux/device.h>
#include <linux/dma-mapping.h>
#include <linux/of.h>
#include <linux/clk.h>
#include <linux/err.h>
#include <linux/pm_runtime.h>
#include <img_mem_man.h>
#include "vha_plat.h"
#include "vha_plat_dt.h"
const struct of_device_id vha_plat_dt_of_ids[] = {
{ .compatible = "img,ax3386-nna" },
//{ .compatible = VHA_PLAT_DT_OF_ID },
{ }
};
static struct heap_config example_heap_configs[] = {
{
.type = IMG_MEM_HEAP_TYPE_UNIFIED,
.options.unified = {
.gfp_type = GFP_KERNEL | __GFP_ZERO,
},
.to_dev_addr = NULL,
},
{
.type = IMG_MEM_HEAP_TYPE_DMABUF,
.to_dev_addr = NULL,
},
{
.type = IMG_MEM_HEAP_TYPE_ANONYMOUS,
.to_dev_addr = NULL,
},
};
struct npu_plat_if {
struct clk *npu_pclk;
struct clk *npu_aclk;
};
static struct npu_plat_if *g_npi;
/*
* IO hooks.
* NOTE: customer may want to use spinlock to avoid
* problems with multi threaded IO access
*/
uint64_t vha_plat_read64(void *addr)
{
return readq((volatile void __iomem *)addr);
}
void vha_plat_write64(void *addr, uint64_t val)
{
writeq(val, (volatile void __iomem *)addr);
}
int vha_plat_dt_hw_init(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
int ret;
uint64_t dma_mask;
dev_dbg(dev, "%s dma_get_mask : %#llx\n", __func__, dma_get_mask(dev));
if (dev->dma_mask) {
dev_info(dev, "%s dev->dma_mask : %p : %#llx\n",
__func__, dev->dma_mask, *dev->dma_mask);
} else {
dev_info(dev, "%s mask unset, setting coherent\n", __func__);
dev->dma_mask = &dev->coherent_dma_mask;
}
/* Try alternative dma_mask setting from device tree */
if (!of_property_read_u64(pdev->dev.of_node, "dma-mask",
(uint64_t *)&dma_mask)) {
dev_info(dev, "%s forcing custom mask from DT : %#llx\n",
__func__, dma_mask);
} else {
/* If alternative mask not defined in
* DT -> "dma-mask" property, use the default one (32bit) */
dma_mask = dma_get_mask(dev);
}
ret = dma_set_mask(dev, dma_mask);
if (ret) {
dev_err(dev, "%s failed to set dma mask\n", __func__);
return ret;
}
/* get clock domain, voltage regulator, set clock rate, etc */
g_npi = devm_kzalloc(&pdev->dev, sizeof(*g_npi), GFP_KERNEL);
if (!g_npi)
return -ENOMEM;
g_npi->npu_pclk = devm_clk_get(&pdev->dev, "pclk");
if (IS_ERR(g_npi->npu_pclk)) {
dev_warn(&pdev->dev, "failed to get npu pclk");
g_npi->npu_pclk == NULL;
}
g_npi->npu_aclk = devm_clk_get(&pdev->dev, "aclk");
if (IS_ERR(g_npi->npu_aclk)) {
dev_warn(&pdev->dev, "failed to get npu aclk");
g_npi->npu_aclk == NULL;
}
return 0;
}
/* return platform global heaps */
void vha_plat_dt_get_heaps(struct heap_config **heap_configs, int *num_heaps)
{
*heap_configs = example_heap_configs;
*num_heaps = sizeof(example_heap_configs)/sizeof(struct heap_config);
}
static int vha_plat_dt_clk_prepare_enable(struct npu_plat_if *npi)
{
int ret;
if (npi->npu_pclk) {
ret = clk_prepare_enable(npi->npu_pclk);
if (ret)
return ret;
}
if (npi->npu_aclk) {
ret = clk_prepare_enable(npi->npu_aclk);
if (ret) {
clk_disable_unprepare(npi->npu_pclk);
return ret;
}
}
return 0;
}
static void vha_plat_dt_clk_disable_unprepare(struct npu_plat_if *npi)
{
if (npi->npu_aclk)
clk_disable_unprepare(npi->npu_aclk);
if (npi->npu_pclk)
clk_disable_unprepare(npi->npu_pclk);
}
void vha_plat_dt_hw_destroy(struct platform_device *pdev)
{
/* Put any vendor related code:
* put clock domain, voltage regulator, etc */
}
int vha_plat_dt_hw_suspend(struct platform_device *pdev)
{
vha_plat_dt_clk_disable_unprepare(g_npi);
return 0;
}
int vha_plat_dt_hw_resume(struct platform_device *pdev)
{
int ret;
ret = vha_plat_dt_clk_prepare_enable(g_npi);
if (ret) {
dev_err(&pdev->dev, "failed to enable npu clock(%d)\n", ret);
return ret;
}
return 0;
}
//MODULE_DEVICE_TABLE(of, vha_plat_dt_of_ids);

122
drivers/nna/vha/platform/vha_plat_thead_light_fpga_c910.c Normal file
View File

@@ -0,0 +1,122 @@
/*!
*****************************************************************************
*
* @File vha_plat_thead_light_fpga_c910.c
* ---------------------------------------------------------------------------
*
* Copyright (C) 2020 Alibaba Group Holding Limited
*
*****************************************************************************/
#include <linux/module.h>
#include <linux/gfp.h>
#include <linux/device.h>
#include <linux/dma-mapping.h>
#include <linux/of.h>
#include <img_mem_man.h>
#include "vha_plat.h"
#include "vha_plat_dt.h"
const struct of_device_id vha_plat_dt_of_ids[] = {
{ .compatible = "img,ax3386-nna" },
//{ .compatible = VHA_PLAT_DT_OF_ID },
{ }
};
static struct heap_config example_heap_configs[] = {
{
.type = IMG_MEM_HEAP_TYPE_UNIFIED,
.options.unified = {
.gfp_type = GFP_KERNEL | __GFP_ZERO,
},
.to_dev_addr = NULL,
},
{
.type = IMG_MEM_HEAP_TYPE_DMABUF,
.to_dev_addr = NULL,
},
{
.type = IMG_MEM_HEAP_TYPE_ANONYMOUS,
.to_dev_addr = NULL,
},
};
/*
* IO hooks.
* NOTE: customer may want to use spinlock to avoid
* problems with multi threaded IO access
*/
uint64_t vha_plat_read64(void *addr)
{
return readq((volatile void __iomem *)addr);
}
void vha_plat_write64(void *addr, uint64_t val)
{
writeq(val, (volatile void __iomem *)addr);
}
int vha_plat_dt_hw_init(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
int ret;
uint64_t dma_mask;
dev_dbg(dev, "%s dma_get_mask : %#llx\n", __func__, dma_get_mask(dev));
if (dev->dma_mask) {
dev_info(dev, "%s dev->dma_mask : %p : %#llx\n",
__func__, dev->dma_mask, *dev->dma_mask);
} else {
dev_info(dev, "%s mask unset, setting coherent\n", __func__);
dev->dma_mask = &dev->coherent_dma_mask;
}
/* Try alternative dma_mask setting from device tree */
if (!of_property_read_u64(pdev->dev.of_node, "dma-mask",
(uint64_t *)&dma_mask)) {
dev_info(dev, "%s forcing custom mask from DT : %#llx\n",
__func__, dma_mask);
} else {
/* If alternative mask not defined in
* DT -> "dma-mask" property, use the default one (32bit) */
dma_mask = dma_get_mask(dev);
}
ret = dma_set_mask(dev, dma_mask);
if (ret) {
dev_err(dev, "%s failed to set dma mask\n", __func__);
return ret;
}
/* Put any vendor related code:
* get clock domain, voltage regulator, set clock rate, etc */
return 0;
}
/* return platform global heaps */
void vha_plat_dt_get_heaps(struct heap_config **heap_configs, int *num_heaps)
{
*heap_configs = example_heap_configs;
*num_heaps = sizeof(example_heap_configs)/sizeof(struct heap_config);
}
void vha_plat_dt_hw_destroy(struct platform_device *pdev)
{
/* Put any vendor related code:
* put clock domain, voltage regulator, etc */
}
int vha_plat_dt_hw_suspend(struct platform_device *pdev)
{
/* This is the place where vendor specific code shall be called:
* eg. turn off voltage regulator/disable power domain */
return 0;
}
int vha_plat_dt_hw_resume(struct platform_device *pdev)
{
/* This is the place where vendor specific code shall be called:
* eg. turn on voltage regulator/enable power domain */
return 0;
}
//MODULE_DEVICE_TABLE(of, vha_plat_dt_of_ids);

750
drivers/nna/vha/single/vha_cnn.c Normal file
View File

@@ -0,0 +1,750 @@
/*
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/slab.h>
#include <linux/device.h>
#include <linux/gfp.h>
#include <linux/moduleparam.h>
#include <linux/delay.h>
#include <uapi/vha.h>
#include "vha_common.h"
#include "vha_plat.h"
#include "vha_regs.h"
static uint32_t cnn_pdump_poll_count = 10000000;
module_param(cnn_pdump_poll_count, uint, 0444);
MODULE_PARM_DESC(cnn_pdump_poll_count,
"PDUMP: Number of times to poll for CNN status");
static bool cnn_preloads_disable;
module_param(cnn_preloads_disable, bool, 0444);
MODULE_PARM_DESC(cnn_preloads_disable,
"Disables CNN preloads");
/*
* submit a command stream to the CNN hardware
* input buffers:
* command
* input
* coeff
* output buffers:
* output
* accum_load
* data:
* none
*/
static int do_cmd_cnn_submit(struct vha_cmd *cmd)
{
int i;
uint32_t val32;
const struct vha_user_cnn_submit_cmd *user_cmd =
(struct vha_user_cnn_submit_cmd *)&cmd->user_cmd;
struct vha_session *session = cmd->session;
struct vha_dev *vha = session->vha;
struct vha_buffer *buf = NULL;
struct vha_onchip_map *onchip_map = NULL;
int ret = -EINVAL;
uint64_t alt_addrs_used = 0;
size_t user_cmd_size;
if (vha->hw_bypass) {
ret = -EAGAIN;
dev_info(vha->dev, "%s skip\n", __func__);
goto out_error;
}
img_pdump_printf("-- CNN_SETUP_BEGIN\n");
/* Wait for the previous kick to be accepted */
if (vha->low_latency != VHA_LL_DISABLED) {
/* Sanity wait for the kick bit to be deasserted */
IOPOLL64_PDUMP(0, 1000, 10, (uint64_t)VHA_CR_OS(CNN_CONTROL_START_EN),
VHA_CR_OS(CNN_CONTROL));
if (cmd->queued &&
vha->low_latency == VHA_LL_SW_KICK)
goto hw_kick;
}
if (vha->pendcmd[VHA_CNN_CMD].cmd != NULL &&
vha->low_latency == VHA_LL_DISABLED) {
dev_err(vha->dev, "%s: trying to submit cnn cmd when hw busy!\n",
__func__);
goto out_error;
}
user_cmd_size = sizeof(*user_cmd);
if (user_cmd->subseg_num > 0)
user_cmd_size += (user_cmd->subseg_num - 1) * sizeof(struct vha_subseg_info);
if (cmd->size != user_cmd_size) {
dev_err(vha->dev, "%s: command buffer wrong size: %zu/%zu",
__func__, cmd->size, sizeof(*user_cmd));
goto out_error;
}
if (!vha_dev_check_hw_capab(vha, user_cmd->expected_ip_capab)) {
ret = -ENODEV;
goto out_error;
}
/* at least CMD and (IN or OUT)*/
if (user_cmd->msg.num_inbufs < 2 ||
/* and maybe TMP and others */
user_cmd->msg.num_bufs > VHA_CORE_MAX_ALT_ADDRS) {
dev_err(vha->dev, "%s: wrong number of bufs: %u,%u\n",
__func__,
user_cmd->msg.num_inbufs, user_cmd->msg.num_bufs);
goto out_error;
}
if (user_cmd->onchipram_map_id != 0) {
onchip_map = idr_find(&session->onchip_maps, user_cmd->onchipram_map_id);
if (!onchip_map) {
dev_warn(vha->dev, "%s: idr_find failed\n", __func__);
}
}
/*
* write buffer address to each register,
* and pdump LDB each of the the input buffers
*/
img_pdump_printf("-- Load inputs\n");
for (i = 0; i < user_cmd->msg.num_bufs; i++) {
uint32_t offset;
uint32_t size;
uint32_t reg;
/* buffer id == 0 means no buffer */
if (user_cmd->msg.data[i] == 0)
continue;
buf = vha_find_bufid(session, user_cmd->msg.data[i]);
if (buf == NULL) {
dev_err(vha->dev, "%s: invalid buffer id:%d\n",
__func__, user_cmd->msg.data[i]);
goto out_error;
}
if (buf->id == user_cmd->cmdbuf) {
/* cmdstream always starts at offset 0 */
if (user_cmd->subseg_info[cmd->subseg_current].cmdbuf_size)
size = cmd->stream_size = user_cmd->subseg_info[cmd->subseg_current].cmdbuf_size;
else
size = cmd->stream_size = buf->size;
offset = user_cmd->subseg_info[cmd->subseg_current].cmdbuf_offset;
if (size == 0) {
dev_err(vha->dev,
"%s: invalid cmdstream size\n",
__func__);
goto out_error;
}
reg = VHA_CR_OS(CNN_CMD_BASE_ADDRESS);
img_pdump_printf("-- Setup command stream\n");
} else {
/*
* offset can be specified for all
* buffers except cmdstream buf
*/
offset = user_cmd->bufoffsets[i-1];
size = user_cmd->bufsizes[i-1];
if (size + offset > buf->size) {
dev_err(vha->dev,
"%s: invalid size+offset: %x+%x > %zx\n",
__func__, size, offset, buf->size);
goto out_error;
}
reg = VHA_CR_OS(CNN_ALT_ADDRESS0)
+ user_cmd->regidx[i-1]
* (VHA_CR_OS(CNN_ALT_ADDRESS1)
- VHA_CR_OS(CNN_ALT_ADDRESS0));
/* record what alt address is in use */
alt_addrs_used |= 1 << user_cmd->regidx[i-1];
#if defined(HW_AX3)
/* Alternative addresses from 8 to 15 are
* located in different place */
if (user_cmd->regidx[i-1] >= 8) {
reg = VHA_CR_OS(CNN_ALT_ADDRESS8)
+ (user_cmd->regidx[i-1] - 8)
* (VHA_CR_OS(CNN_ALT_ADDRESS1)
- VHA_CR_OS(CNN_ALT_ADDRESS0));
}
alt_addrs_used |= buf->req_type <<
(VHA_CR_OS(CNN_ALT_ADDRESS_USED_ALT_ADDR0_BUF_TYPE_SHIFT) +
user_cmd->regidx[i-1]);
#elif defined(HW_AX2)
if (user_cmd->regidx[i-1] > 8) {
dev_err(vha->dev,
"%s: extended alternative addresses not supported!\n",
__func__);
goto out_error;
}
#endif
}
/* pdump the input buffers (not filled by the hw),
* try to cache buffers filled by SW,
* to avoid unnecessary LDBs */
if (i < user_cmd->msg.num_inbufs &&
!(buf->status == VHA_BUF_FILLED_BY_HW))
vha_pdump_ldb_buf(session, PDUMP_PRM,
buf, offset, size,
buf->status == VHA_BUF_FILLED_BY_SW);
vha_dump_digest(session, buf, cmd);
/*
* write to all of the index registers.
* in no-MMU mode, write phys address of a contig buffer.
* in MMU mode, write virt address of buffer.
* If onchip_map selected, use different virt address of buffer
*/
if (onchip_map != NULL && onchip_map->bufid == buf->id)
IOWRITE64_PDUMP(onchip_map->devvirt + offset, reg);
else
IOWRITE_PDUMP_BUFADDR(session, buf, offset, reg);
if (vha_buf_needs_flush(session, buf->id))
img_mem_sync_cpu_to_device(session->mem_ctx, buf->id);
}
if (!cnn_preloads_disable) {
/* Inform the hw what alt addresses are in use,
* so the command decoder can prefetch */
img_pdump_printf("-- Setup CNN prefetch register\n");
IOWRITE64_PDUMP(alt_addrs_used, VHA_CR_OS(CNN_ALT_ADDRESS_USED));
}
/* write the stream size only */
ret = 0;
if (vha->pendcmd[VHA_CNN_CMD].cmd) {
vha->queuedcmd[VHA_CNN_CMD].cmd = cmd;
cmd->queued = true;
vha->stats.cnn_kicks_queued++;
img_pdump_printf("-- CNN already kicked queueing!\n");
dev_dbg(vha->dev, "%s: -> kicked:%p queueing:%p\n",
__func__, vha->pendcmd[VHA_CNN_CMD].cmd, cmd);
if (vha->low_latency == VHA_LL_SW_KICK)
return ret;
}
hw_kick:
/* Change mmu context */
ret = vha_mmu_setup(cmd->session);
if (ret) {
dev_err(vha->dev,
"%s: Error during MMU setup!\n", __func__);
goto out_error;
}
/* Setup memory stuff */
vha_dev_mh_setup(vha, session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].hw_id, NULL);
/* Prepare debug buffer registers */
vha_dbg_prepare_hwbufs(session, cmd, NULL);
/* Setup cnn hw watchdog before kicking the hw */
{
uint64_t cycles, mode;
ret = vha_dev_hwwdt_calculate(vha, cmd, &cycles, &mode);
if (!ret)
vha_dev_hwwdt_setup(session->vha, cycles, mode);
else if (ret != -EIO) {
dev_err(vha->dev,
"%s: can't obtain HWWDT info!\n",
__func__);
goto out_error;
}
}
if (CMD_EXEC_ON_HW(cmd)) {
cmd->in_hw = true;
if (!cmd->queued)
vha->pendcmd[VHA_CNN_CMD].cmd = cmd;
}
#ifdef CONFIG_VHA_DUMMY_SIMULATE_HW_PROCESSING_TIME
/* Mark kick for dummy driver */
cmd->dummy_kicked = true;
#endif
/* Consider this cmd as kicked. */
vha->pri_q_counters[cmd->user_cmd.priority]--;
cmd->subseg_current++;
ret = 0;
/* Setup kick info */
val32 = vha_dev_kick_prepare(vha, cmd,
session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].hw_id);
img_pdump_printf("-- CNN_SETUP_END\n");
/* Remember the time cnn is kicked */
GETNSTIMEOFDAY(&cmd->hw_proc_start);
vha->stats.hw_proc_start = cmd->hw_proc_start;
/* Need to generate proper pdump */
if (cmd->queued &&
vha->low_latency == VHA_LL_SW_KICK) {
/* Do not write to pdump
* this needs to be done after irq POL*/
IOWRITE64(vha->reg_base, VHA_CR_OS(CNN_CONTROL), val32);
dev_dbg(vha->dev, "%s: CNN kick queued (%p)!\n",
__func__, cmd);
cmd->queued = false;
} else {
img_pdump_printf("-- CNN_KICK_BEGIN\n");
img_pdump_printf("-- CNN kick!\n");
IOWRITE64_PDUMP(val32, VHA_CR_OS(CNN_CONTROL));
dev_dbg(vha->dev, "%s: CNN kick %s (%p)!\n",
__func__, cmd->queued ? "queued" : "", cmd);
img_pdump_printf("-- CNN_KICK_END\n");
}
vha->stats.cnn_kicks++;
/* notify any observers of the submit event */
if (vha_observers.submitted)
vha_observers.submitted(vha->id, session->id, cmd->user_cmd.cmd_id,
(cmd->subseg_current == VHA_CMD_SUBSEG_NUM(cmd)),
cmd->user_cmd.priority);
out_error:
if (ret != 0) {
/* Consider this cmd as kicked for errors too. */
vha->pri_q_counters[cmd->user_cmd.priority]--;
cmd->subseg_current++;
}
return ret;
}
/*
* append a string to the pdump TXT file
* buffers:
* none
* data:
* string to be printed
*/
static int do_cmd_cnn_pdump_msg(const struct vha_cmd *cmd)
{
const struct vha_user_cmd *user_cmd = &cmd->user_cmd;
struct vha_session *session = cmd->session;
struct vha_dev* vha = session->vha;
int ret = 0;
if (user_cmd->num_inbufs != 0 || user_cmd->num_bufs != 0) {
dev_err(session->vha->dev, ">0 buffers in cmd is wrong\n");
ret = -EINVAL;
}
/* remember the pdump message may not be null terminated */
img_pdump_printf("%.*s\n", (int)cmd->size, (char *)user_cmd->data);
return ret;
}
/*
* Simple procedure that generates watchdog interrupt
*/
void vha_cnn_start_calib(struct vha_dev *vha)
{
uint64_t clk;
uint32_t start;
/* Setup hw watchdog before kicking the hw */
vha_dev_hwwdt_setup(vha, vha->calibration_cycles, 0);
/* Disabling command decoder, so we can generate wdt interrupt,
* without providing any buffer address */
clk = IOREAD64(vha->reg_base, VHA_CR_CLK_CTRL0);
VHA_CR_CLEARBITS(clk, CLK_CTRL0, CNN_CMD);
IOWRITE64(vha->reg_base, VHA_CR_CLK_CTRL0, clk);
/* To be sure the cmd clock has switched off*/
udelay(100);
/* Enable MMU bypass */
IOWRITE64_PDUMP(VHA_CR_OS(MMU_CTRL_BYPASS_EN),
VHA_CR_OS(MMU_CTRL));
/* Set minimal command stream size */
start = (2048/32-1) << VHA_CR_OS(CNN_CONTROL_CMD_SIZE_MIN1_SHIFT);
start |= VHA_CR_OS(CNN_CONTROL_START_EN);
/* write the START bit */
IOWRITE64(vha->reg_base, VHA_CR_OS(CNN_CONTROL), start);
/* Remember the time cnn is kicked */
GETNSTIMEOFDAY(&vha->stats.hw_proc_start);
}
void vha_cnn_update_stats(struct vha_dev *vha)
{
vha->stats.cnn_last_proc_us =
vha->stats.last_proc_us;
vha->stats.cnn_total_proc_us +=
vha->stats.last_proc_us;
if (vha->stats.cnn_kicks) {
uint64_t avg = vha->stats.cnn_total_proc_us;
do_div(avg, vha->stats.cnn_kicks);
vha->stats.cnn_avg_proc_us = avg;
}
#if defined(HW_AX2)
vha->stats.cnn_last_cycles =
IOREAD64(vha->reg_base, VHA_CR_CNN_WDT_TIMER);
#elif defined(HW_AX3)
vha->stats.cnn_last_cycles =
IOREAD64(vha->reg_base, VHA_CR_OS(CNN_PERFORMANCE));
#endif
if (vha->stats.cnn_last_cycles && vha->freq_khz) {
uint64_t est_proc_us = 1000UL * vha->stats.cnn_last_cycles;
do_div(est_proc_us, vha->freq_khz);
vha->stats.cnn_last_est_proc_us = est_proc_us;
}
vha->stats.cnn_total_cycles += vha->stats.cnn_last_cycles;
if (vha->stats.cnn_kicks &&
vha->stats.cnn_total_cycles && vha->freq_khz) {
uint64_t avg = 1000UL * vha->stats.cnn_total_cycles;
do_div(avg, vha->stats.cnn_kicks);
do_div(avg, vha->freq_khz);
vha->stats.cnn_avg_est_proc_us = avg;
}
}
/*
* a command has completed. sent notification to user
*/
void vha_cnn_cmd_completed(struct vha_cmd *cmd, int status)
{
struct vha_session *session = cmd->session;
struct vha_dev* vha = session->vha;
struct vha_rsp *rsp = NULL;
int i;
struct vha_user_cnn_submit_rsp * cnn_submit_rsp = NULL;
const struct vha_user_cmd *user_cmd = &cmd->user_cmd;
switch (user_cmd->cmd_type) {
case VHA_CMD_CNN_SUBMIT:
{
size_t mem_usage;
/* allocate sufficient space for the response */
size_t sz = sizeof(*rsp)
+ sizeof(struct vha_user_cnn_submit_rsp)
- sizeof(struct vha_user_rsp);
uint32_t status_mask;
uint32_t ready_mask;
uint32_t cmpl_val = VHA_CR_OS(VHA_EVENT_STATUS_VHA_CNN0_COMPLETE_EN);
#if defined(HW_AX2)
/* status change: wait for any status change:
* WDT, MMU_PF, ERROR, COMPLETE
*/
status_mask = 0xffffffff;
ready_mask = 0xffffffff;
#elif defined(HW_AX3)
/* status mask: wait for a status change: either ERROR, COMPLETE:
* note that, unlike the live driver, pdump will ignore the MMU_PF,
* which will have to be detected by the WDT
*/
status_mask = VHA_CR_OS(VHA_EVENT_STATUS_VHA_ERROR_CLRMSK)
| VHA_CR_OS(VHA_EVENT_STATUS_VHA_CNN0_COMPLETE_CLRMSK);
ready_mask = VHA_CR_OS(VHA_EVENT_STATUS_VHA_READY_CLRMSK);
/* Ignore PARITY when waiting for status change */
status_mask &= VHA_CR_OS(VHA_EVENT_STATUS_PARITY_CLRMSK);
#ifdef VHA_SCF
if (session->vha->hw_props.supported.parity &&
!session->vha->parity_disable) {
/* If complete bit is set then parity bit must be set as well ! */
cmpl_val |= VHA_CR_OS(VHA_EVENT_STATUS_PARITY_EN);
}
#else
/* Ignore PARITY, so that non-SCF pdump may work with SC CSIM */
ready_mask &= VHA_CR_OS(VHA_EVENT_STATUS_PARITY_CLRMSK);
#endif
#endif
rsp = kzalloc(sz, GFP_KERNEL);
if (rsp == NULL) {
session->oom = true;
return;
}
cnn_submit_rsp = (struct vha_user_cnn_submit_rsp*)&rsp->user_rsp;
rsp->size = sizeof(struct vha_user_cnn_submit_rsp);
if (session->vha->hw_bypass) {
session->vha->hw_bypass--;
break;
}
img_pdump_printf("-- CNN_WAIT_BEGIN\n");
/* pdump POL for status change
* count=cnn_pdump_poll_count, delay=1000cycles
*/
img_pdump_printf("-- Wait for any CNN status\n"
"POL :REG:%#x 0 %#x 3 %u 1000\n",
VHA_CR_OS(VHA_EVENT_STATUS),
status_mask,
cnn_pdump_poll_count);
/* quick pdump POL for the status complete flag only:
* count=1, delay=10cycles
*/
img_pdump_printf("-- Check for CNN_COMPLETE flag only\n"
"POL :REG:%#x %#x 0x%x 0 1 10\n",
VHA_CR_OS(VHA_EVENT_STATUS),
cmpl_val,
ready_mask);
#ifdef VHA_SCF
if (session->vha->hw_props.supported.parity &&
!session->vha->parity_disable) {
/* Check CNN_STATUS parity */
uint32_t cnn_status = VHA_CR_SETBITS_OS(CNN_STATUS,
STREAM_COUNT, 1);
cnn_status |= VHA_CR_SETBITS_OS(CNN_STATUS,
PARITY, 1);
img_pdump_printf("-- Check for CNN_STATUS parity\n"
"POL :REG:%#x %#x 0xffffffff 0 1 10\n",
VHA_CR_OS(CNN_STATUS), cnn_status);
}
#endif
/* quick pdump POL for AXI errors:
* count=1, delay=10cycles
*/
img_pdump_printf("-- Post check of AXI status\n"
"POL :REG:%#x 0 0xffffffff 0 1 10\n",
VHA_CR_ACE_STATUS);
/* We do clear interrupts in the irq handler,
* but this is not recorded into pdump because
* of the irq context, so do it here */
img_pdump_printf("-- Clear CNN events\n"
"WRW64 :REG:%#x %#x\n",
VHA_CR_OS(VHA_EVENT_CLEAR),
VHA_CR_OS(VHA_EVENT_CLEAR_VHA_CNN0_COMPLETE_EN) |
VHA_CNN_ERR_EVNTS);
/* Try to flush hw debug buffers first
* - this does pdump SAB when proper checkpoint is set */
vha_dbg_flush_hwbufs(session, 1, 0);
/* pdump SAB for each of the output buffers */
img_pdump_printf("-- Save outputs\n");
for (i = user_cmd->num_inbufs; i < user_cmd->num_bufs; i++) {
struct vha_buffer *buf;
struct vha_user_cnn_submit_cmd *msg =
container_of(user_cmd,
struct vha_user_cnn_submit_cmd,
msg);
uint32_t offset;
uint32_t size;
buf = vha_find_bufid(session, user_cmd->data[i]);
if (buf == NULL) {
dev_err(session->vha->dev,
"%s: invalid buffer id:%d\n",
__func__, user_cmd->data[i]);
continue;
}
if (buf->id == msg->cmdbuf) {
offset = 0;
size = buf->size;
} else {
offset = msg->bufoffsets[i-1];
size = msg->bufsizes[i-1];
}
vha_pdump_sab_buf(session, PDUMP_RES,
buf, offset, size);
/* Update status, do not signal fence yet,
* it's is done explicitly below, after cache invalidation */
vha_set_buf_status(session, buf->id, VHA_BUF_FILLED_BY_HW,
VHA_SYNC_NONE, false);
if (vha_buf_needs_inval(session, buf->id) && !status)
img_mem_sync_device_to_cpu(session->mem_ctx, buf->id);
#ifdef KERNEL_DMA_FENCE_SUPPORT
img_mem_signal_fence(session->mem_ctx, buf->id);
#endif
vha_dump_digest(session, buf, cmd);
}
if (session->vha->low_latency == VHA_LL_SW_KICK) {
struct vha_cmd *cmd =
session->vha->queuedcmd[VHA_CNN_CMD].cmd;
if (cmd && cmd->queued) {
/* Setup kick info */
uint64_t val = vha_dev_kick_prepare(session->vha, cmd,
session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].hw_id);
img_pdump_printf("-- CNN kick (queued)!\n");
img_pdump_printf("WRW64 :REG:%#x %#llx\n",
VHA_CR_OS(CNN_CONTROL), val);
}
}
img_pdump_printf("-- CNN_WAIT_END\n");
img_mem_get_usage(session->mem_ctx, NULL, &mem_usage);
/* send out an event when submit is complete */
if (vha_observers.completed)
vha_observers.completed(
session->vha->id,
session->id,
user_cmd->cmd_id,
status,
session->vha->stats.cnn_last_cycles,
mem_usage,
user_cmd->priority);
/* post some metrics about the hw to user space */
#ifdef MEM_USAGE_LAST_METRICS_ARE_AVAILABLE
cnn_submit_rsp->mem_usage = mem_usage;
#else
cnn_submit_rsp->mem_usage = ~0;
#endif
cnn_submit_rsp->last_proc_us = cmd->proc_us;
cnn_submit_rsp->hw_cycles = cmd->hw_cycles;
dev_dbg(session->vha->dev, "%s: %p, hw_cycles %llx\n", __func__,
cmd, session->vha->stats.cnn_last_cycles);
if (session->vha->stats.cnn_last_cycles > (uint32_t)~0)
dev_warn(session->vha->dev,
"%s: hw_cycles %llx exceeds 32bit limit\n",
__func__,
session->vha->stats.cnn_last_cycles);
break;
}
case VHA_CMD_CNN_PDUMP_MSG:
default:
/* allocate space for standard response */
rsp = kzalloc(sizeof(*rsp), GFP_KERNEL);
if (rsp == NULL) {
session->oom = true;
return;
}
rsp->size = sizeof(rsp->user_rsp);
break;
}
if (user_cmd->flags & VHA_CMDFLAG_NOTIFY) {
rsp->user_rsp.cmd_id = cmd->user_cmd.cmd_id;
rsp->user_rsp.err_no = session->vha->hw_bypass ? 0 : status;
cmd->rsp = rsp;
} else
kfree(rsp);
}
/*
* Perform a command, as requested by user.
* note: this function is called with vha_dev.lock == locked
*/
int vha_do_cnn_cmd(struct vha_cmd *cmd)
{
struct vha_session *session = cmd->session;
const struct vha_user_cmd *user_cmd = &cmd->user_cmd;
int status = -EINVAL;
dev_dbg(session->vha->dev,
"CNN command: id:%x type:%x nin:%x nbufs:%x\n",
user_cmd->cmd_id, user_cmd->cmd_type,
user_cmd->num_inbufs, user_cmd->num_bufs);
#if 0
print_hex_dump_debug("VHA CMD: ", DUMP_PREFIX_NONE, 4, 4,
user_cmd, ALIGN(cmd->size, 4), false);
#endif
switch (user_cmd->cmd_type) {
case VHA_CMD_CNN_SUBMIT:
status = do_cmd_cnn_submit(cmd);
#ifdef CONFIG_VHA_DUMMY_SIMULATE_HW_PROCESSING_TIME
if (cmd->dummy_kicked) {
struct vha_dev *vha = cmd->session->vha;
const struct vha_user_cnn_submit_cmd *cnn_user_cmd =
(struct vha_user_cnn_submit_cmd *)&cmd->user_cmd;
uint32_t estimated_cycles = cnn_user_cmd->estimated_cycles;
if (estimated_cycles == 0)
estimated_cycles = VHA_DUMMY_HW_PROCESSING_TIME_CYCLES;
cmd->dummy_exec_time = (estimated_cycles / (vha->freq_khz / 1000));
schedule_delayed_work(&vha->dummy_dwork,
usecs_to_jiffies(cmd->dummy_exec_time));
cmd->dummy_kicked = false;
}
#endif
break;
case VHA_CMD_CNN_PDUMP_MSG:
status = do_cmd_cnn_pdump_msg(cmd);
default:
break;
}
/*
* Immediately send notification to user if not using hw at all
* or submitting failed.
*/
if (!CMD_EXEC_ON_HW(cmd) || status) {
vha_cnn_cmd_completed(cmd, status);
vha_cmd_notify(cmd);
return 1;
}
return 0;
}
void vha_cnn_dump_status(struct vha_dev *vha)
{
struct device *dev = vha->dev;
dev_err(dev, " CNN_STATUS:%llx ",
IOREAD64(vha->reg_base,
VHA_CR_OS(CNN_STATUS)));
#ifdef HW_AX2
dev_err(dev, " CNN_WDT_COMPAREMATCH:%llx ",
IOREAD64(vha->reg_base,
VHA_CR_CNN_WDT_COMPAREMATCH));
dev_err(dev, " CNN_WDT_TIMER:%llx ",
IOREAD64(vha->reg_base,
VHA_CR_CNN_WDT_TIMER));
#endif
dev_err(dev, " CNN_MEM_WDT_COMPAREMATCH:%llx ",
IOREAD64(vha->reg_base,
VHA_CR_CNN_MEM_WDT_COMPAREMATCH));
dev_err(dev, " CNN_MEM_WDT_TIMER:%llx ",
IOREAD64(vha->reg_base,
VHA_CR_CNN_MEM_WDT_TIMER));
dev_err(dev, " BIF_OUTSTANDING_READ:%llx\n",
IOREAD64(vha->reg_base,
VHA_CR_BIF_OUTSTANDING_READ));
}

1581
drivers/nna/vha/single/vha_dev.c Normal file
View File

File diff suppressed because it is too large Load Diff

190
drivers/nna/vha/single/vha_dev_ax2.c Normal file
View File

@@ -0,0 +1,190 @@
/*
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/device.h>
#include <linux/moduleparam.h>
#include "vha_common.h"
#include "vha_plat.h"
#include "vha_regs.h"
static long cnn_wdt_cycles = VHA_CORE_WDT_CYCLES;
module_param(cnn_wdt_cycles, long, 0444);
MODULE_PARM_DESC(cnn_wdt_cycles,
"CNN hw watchdog expiration cycles, -1=use estimated cycles, 0=disable watchdog, >0=predefined");
static uint32_t cnn_wdt_cycles_margin = 40;
module_param(cnn_wdt_cycles_margin, uint, 0444);
MODULE_PARM_DESC(cnn_wdt_cycles_margin,
"CNN estimated hw watchdog percentage overhead. default:40% additional margin added");
void vha_dev_mh_setup(struct vha_dev *vha, int ctx_id, struct vha_mh_config_regs *regs)
{
uint64_t val64 = 0;
uint8_t burst = ilog2(VHA_CORE_MH_MAX_BURST_LENGTH/32);
WARN_ON(burst & ~VHA_CR_MH_CONTROL_MAX_BURST_LENGTH_MASK);
val64 |= VHA_CR_SETBITS(CNN_CMD_MH_CONTROL,
MAX_BURST_LENGTH, burst);
val64 |= VHA_CR_SETBITS(CNN_CMD_MH_CONTROL,
GPU_PIPE_COHERENT, VHA_CORE_MH_GPU_PIPE_COHERENT_TYPE);
val64 |= VHA_CR_SETBITS(CNN_CMD_MH_CONTROL,
SLC_CACHE_POLICY, VHA_CORE_MH_SLC_CACHE_POLICY_TYPE);
val64 |= VHA_CR_SETBITS(CNN_CMD_MH_CONTROL,
PERSISTENCE, VHA_CORE_MH_PERSISTENCE_PRIO);
img_pdump_printf("-- CNN mem hierarchy setup CTXT_PASID:%d\n", ctx_id);
val64 |= VHA_CR_SETBITS(CNN_CMD_MH_CONTROL,
CTXT_PASID, ctx_id);
/* Note: CMD reg has different layout than IBUF,CBUF,ABUFF,OUPACK */
IOWRITE64_PDUMP(val64, VHA_CR_CNN_CMD_MH_CONTROL);
val64 = 0;
val64 |= VHA_CR_SETBITS(CNN_IBUF_MH_CONTROL,
MAX_BURST_LENGTH, burst);
val64 |= VHA_CR_SETBITS(CNN_IBUF_MH_CONTROL,
GPU_PIPE_COHERENT, VHA_CORE_MH_GPU_PIPE_COHERENT_TYPE);
val64 |= VHA_CR_SETBITS(CNN_IBUF_MH_CONTROL,
PERSISTENCE, VHA_CORE_MH_PERSISTENCE_PRIO);
IOWRITE64_PDUMP(val64, VHA_CR_CNN_IBUF_MH_CONTROL);
IOWRITE64_PDUMP(val64, VHA_CR_CNN_CBUF_MH_CONTROL);
IOWRITE64_PDUMP(val64, VHA_CR_CNN_ABUF_MH_CONTROL);
IOWRITE64_PDUMP(val64, VHA_CR_CNN_OUTPACK_MH_CONTROL);
IOWRITE64_PDUMP(val64, VHA_CR_CNN_ELEMENTOPS_MH_CONTROL);
}
void vha_dev_hwwdt_setup(struct vha_dev *vha, uint64_t cycles, uint64_t mode)
{
if (!mode)
mode = VHA_CR_CNN_WDT_CTRL_CNN_WDT_CTRL_KICK_PASS;
dev_dbg(vha->dev, "%s: cycles:%llx mode:%llx\n", __func__, cycles, mode);
/* Note: Do not pdump the main watchdog as it may trigger
* during memory latency/stalling testing */
if (cycles) {
IOWRITE64(vha->reg_base, VHA_CR_CNN_WDT_COMPAREMATCH,
cycles & VHA_CR_CNN_WDT_COMPAREMATCH_MASKFULL);
IOWRITE64(vha->reg_base, VHA_CR_CNN_WDT_CTRL,
mode & VHA_CR_CNN_WDT_CTRL_CNN_WDT_CTRL_MASK);
} else {
IOWRITE64(vha->reg_base, VHA_CR_CNN_WDT_CTRL,
VHA_CR_CNN_WDT_CTRL_CNN_WDT_CTRL_NONE);
}
/* Clear timer value just for sanity */
IOWRITE64(vha->reg_base, VHA_CR_CNN_WDT_TIMER, 0);
/* Note: We are not enabling MEM_WDT because it will not detect
* issues due to the BIF/MMU or the customers memory fabric.
* We could in theory enable this in customer systems,
* but there is always a risk that it would result in false negatives
* if there memory latency went very high temporarily.
* HW team set this watchdog externally */
#if 0
IOWRITE64(vha->reg_base, VHA_CR_CNN_MEM_WDT_COMPAREMATCH, 0xfffff);
IOWRITE64(vha->reg_base, VHA_CR_CNN_MEM_WDT_CTRL,
VHA_CR_CNN_MEM_WDT_CTRL_CNN_MEM_WDT_CTRL_KICK_PASS);
/* Clear timer value */
IOWRITE64(vha->reg_base + VHA_CR_CNN_MEM_WDT_TIMER, 0);
#endif
}
int vha_dev_hwwdt_calculate(struct vha_dev *vha, struct vha_cmd *cmd,
uint64_t *cycles, uint64_t *mode)
{
const struct vha_user_cnn_submit_cmd *user_cmd =
(struct vha_user_cnn_submit_cmd *)&cmd->user_cmd;
if (!cycles || !mode)
return -EINVAL;
if (user_cmd && user_cmd->estimated_cycles && cnn_wdt_cycles == -1) {
/* allow 40%, by default, above the estimated cycle count.
* Clamp at uint32_t maximum */
uint64_t wdt_cycles = user_cmd->estimated_cycles;
uint64_t margin = wdt_cycles * cnn_wdt_cycles_margin;
do_div(margin, 100UL);
dev_dbg(vha->dev,
"%s: estimated wdt cycles:%llx + margin:%llx\n",
__func__, wdt_cycles, margin);
wdt_cycles += margin;
if (wdt_cycles > 0xffffffff)
wdt_cycles = 0xffffffff;
/* estimated cycle is per segment */
*cycles = wdt_cycles;
*mode = VHA_CR_CNN_WDT_CTRL_CNN_WDT_CTRL_KICK;
} else {
/* default value is per pass.
* If default is 0 cycles it disables the watchdog */
*cycles = cnn_wdt_cycles;
*mode = VHA_CR_CNN_WDT_CTRL_CNN_WDT_CTRL_KICK_PASS;
}
vha->wdt_mode = *mode;
return 0;
}
int vha_dev_prepare(struct vha_dev *vha)
{
/* Nothing to do */
return 0;
}
void vha_dev_setup(struct vha_dev *vha)
{
vha->is_ready = true;
}
void vha_dev_wait(struct vha_dev *vha)
{
/* Nothing to do */
}
uint32_t vha_dev_kick_prepare(struct vha_dev *vha,
struct vha_cmd *cmd, int ctx_id)
{
/* write to the START bit */
uint32_t val = (min(2048U, cmd->stream_size)/32-1)
<< VHA_CR_OS(CNN_CONTROL_CMD_SIZE_MIN1_SHIFT);
val |= VHA_CR_OS(CNN_CONTROL_START_EN);
return val;
}

207
drivers/nna/vha/single/vha_dev_ax3.c Normal file
View File

@@ -0,0 +1,207 @@
/*
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/device.h>
#include <linux/moduleparam.h>
#include "vha_common.h"
#include "vha_plat.h"
#include "vha_regs.h"
static uint32_t os_priority = _OSID_;
module_param(os_priority, uint, 0444);
MODULE_PARM_DESC(os_priority, "Kick priority for this driver instance: <0,3>");
static uint32_t prio_limits;
module_param(prio_limits, uint, 0444);
MODULE_PARM_DESC(prio_limits, "Priority limits. Valid for OS0 only. See TRM");
static uint32_t hl_wdt_cycles = VHA_CORE_WDT_CYCLES;
module_param(hl_wdt_cycles, uint, 0444);
MODULE_PARM_DESC(hl_wdt_cycles, "High level watchdog cycles");
static uint32_t hl_wdt_mode = 1;
module_param(hl_wdt_mode, uint, 0444);
MODULE_PARM_DESC(hl_wdt_mode, "High level watchdog mode: 1-pass; 2-layer group. See TRM");
void vha_dev_mh_setup(struct vha_dev *vha, int ctx_id, struct vha_mh_config_regs *regs)
{
uint64_t val64 = 0;
val64 |= VHA_CR_SETBITS_OS(CNN_PRELOAD_CONTROL, CBUF_N_REQS,
VHA_CR_CNN_PRELOAD_CTRL_N_64);
/* Setup preload for MMM */
val64 |= VHA_CR_SETBITS_OS(CNN_PRELOAD_CONTROL, MMM_RD_N_REQS, VHA_CR_CNN_PRELOAD_CTRL_N_256);
val64 |= VHA_CR_SETBITS_OS(CNN_PRELOAD_CONTROL, MMM_WR_N_REQS, VHA_CR_CNN_PRELOAD_CTRL_N_256);
IOWRITE64_PDUMP(val64, VHA_CR_OS(CNN_PRELOAD_CONTROL));
}
void vha_dev_hwwdt_setup(struct vha_dev *vha, uint64_t cycles, uint64_t mode)
{
img_pdump_printf("-- Setup High level watchdog\n");
IOWRITE64_PDUMP((cycles & VHA_CR_CNN_HL_WDT_COMPAREMATCH_MASKFULL),
VHA_CR_CNN_HL_WDT_COMPAREMATCH);
IOWRITE64_PDUMP(hl_wdt_mode,
VHA_CR_CNN_HL_WDT_CTRL);
IOWRITE64_PDUMP(0, VHA_CR_CNN_HL_WDT_TIMER);
/* Setup memory watchdog */
IOWRITE64(vha->reg_base, VHA_CR_CNN_MEM_WDT_COMPAREMATCH, VHA_CORE_MEM_WDT_CYCLES);
IOWRITE64(vha->reg_base, VHA_CR_CNN_MEM_WDT_CTRL,
VHA_CR_CNN_MEM_WDT_CTRL_CNN_MEM_WDT_CTRL_KICK_PASS);
IOWRITE64(vha->reg_base, VHA_CR_CNN_MEM_WDT_TIMER, 0);
}
int vha_dev_hwwdt_calculate(struct vha_dev *vha, struct vha_cmd *cmd,
uint64_t *cycles, uint64_t *mode)
{
if (!cycles || !mode)
return -EINVAL;
return -EIO;
}
int vha_dev_prepare(struct vha_dev *vha)
{
/* Enable core events */
img_pdump_printf("-- Enable CORE events\n");
IOWRITE64_PDUMP(VHA_CORE_EVNTS, VHA_CR_OS(VHA_EVENT_ENABLE));
img_pdump_printf("-- Clear CORE events\n");
IOWRITE64_PDUMP(VHA_CORE_EVNTS, VHA_CR_OS(VHA_EVENT_CLEAR));
return 0;
}
void vha_dev_setup(struct vha_dev *vha)
{
uint64_t val64;
vha_dev_hwwdt_setup(vha, hl_wdt_cycles, 0);
if (prio_limits) {
img_pdump_printf("-- Set priority limits\n");
IOWRITE64_PDUMP(prio_limits, VHA_CR_CNN_CMD_PRIORITY_LIMITS);
}
img_pdump_printf("-- MMU set virtual address range0:%#llx-%#llx\n",
IMG_MEM_VA_HEAP1_BASE, IMG_MEM_VA_HEAP1_SIZE);
val64 = (uint64_t)vha->mmu_page_size <<
VHA_CR_MMU_PAGE_SIZE_RANGE_ONE_PAGE_SIZE_SHIFT;
val64 |= VHA_CR_ALIGN_SETBITS(MMU_PAGE_SIZE_RANGE_ONE,
BASE_ADDR, IMG_MEM_VA_HEAP1_BASE);
val64 |= VHA_CR_ALIGN_SETBITS(MMU_PAGE_SIZE_RANGE_ONE,
END_ADDR, (IMG_MEM_VA_HEAP1_BASE + IMG_MEM_VA_HEAP1_SIZE));
IOWRITE64_PDUMP(val64, VHA_CR_MMU_PAGE_SIZE_RANGE_ONE);
img_pdump_printf("-- MMU set virtual address range1:%#llx-%#llx\n",
IMG_MEM_VA_HEAP2_BASE, IMG_MEM_VA_HEAP2_SIZE);
val64 = (uint64_t)vha->mmu_page_size <<
VHA_CR_MMU_PAGE_SIZE_RANGE_TWO_PAGE_SIZE_SHIFT ;
val64 |= VHA_CR_ALIGN_SETBITS(MMU_PAGE_SIZE_RANGE_TWO,
BASE_ADDR, IMG_MEM_VA_HEAP2_BASE);
val64 |= VHA_CR_ALIGN_SETBITS(MMU_PAGE_SIZE_RANGE_TWO,
END_ADDR, (IMG_MEM_VA_HEAP2_BASE + IMG_MEM_VA_HEAP2_SIZE));
IOWRITE64_PDUMP(val64, VHA_CR_MMU_PAGE_SIZE_RANGE_TWO);
}
void vha_dev_wait(struct vha_dev *vha)
{
uint32_t ready_val = VHA_CR_OS(VHA_EVENT_STATUS_VHA_READY_EN);
uint32_t ready_mask = 0xffffffff;
/* Ignore PARITY when waiting for status change */
uint32_t status_mask = VHA_CR_OS(VHA_EVENT_STATUS_PARITY_CLRMSK);
#ifdef VHA_SCF
if (vha->hw_props.supported.parity &&
!vha->parity_disable) {
/* If READY bit is set then parity bit must be set as well ! */
ready_val |= VHA_CR_OS(VHA_EVENT_STATUS_PARITY_EN);
}
#else
/* Ignore PARITY, so that non-SCF pdump may work with SC CSIM */
ready_mask &= VHA_CR_OS(VHA_EVENT_STATUS_PARITY_CLRMSK);
#endif
/* Wait for READY interrupt as well
* pdump POL for any status flag:
* count=100, delay=100cycles
*/
img_pdump_printf("-- Wait for any CORE status change\n"
"POL :REG:%#x 0 %#x 3 1000 1000\n",
VHA_CR_OS(VHA_EVENT_STATUS), status_mask);
/* quick pdump POL for the status READY flag only:
* count=1, delay=10cycles
*/
img_pdump_printf("-- Check for READY flag only\n"
"POL :REG:%#x %#x %#x 0 1 10\n",
VHA_CR_OS(VHA_EVENT_STATUS),
ready_val, ready_mask);
/* We do clear interrupts in the irq handler,
* but this is not recorded into pdump because
* of the irq context, so do it here */
img_pdump_printf("-- Clear CORE events\n"
"WRW64 :REG:%#x %#x\n",
VHA_CR_OS(VHA_EVENT_CLEAR),
VHA_CR_OS(VHA_EVENT_CLEAR_VHA_READY_EN) |
VHA_CR_OS(VHA_EVENT_CLEAR_VHA_ERROR_EN) |
VHA_CR_OS(VHA_EVENT_CLEAR_VHA_HL_WDT_EN));
}
uint32_t vha_dev_kick_prepare(struct vha_dev *vha,
struct vha_cmd *cmd, int ctx_id)
{
/* write to the START bit */
uint32_t val32 = (min(2048U, cmd->stream_size)/32-1)
<< VHA_CR_OS(CNN_CONTROL_CMD_SIZE_MIN1_SHIFT);
val32 |= VHA_CR_OS(CNN_CONTROL_START_EN);
/* This is odd, hw uses two contexts, we provide the base one,
* but the other is always used in pair */
img_pdump_printf("-- CNN setup CTXT_PASID:%d PRIO:%d\n",
ctx_id, os_priority);
val32 |= VHA_CR_SETBITS_OS(CNN_CONTROL,
CTXT_PASID, ctx_id);
val32 |= VHA_CR_SETBITS_OS(CNN_CONTROL,
PRIORITY, os_priority);
return val32;
}

241
drivers/nna/vha/single/vha_mmu.c Normal file
View File

@@ -0,0 +1,241 @@
/*
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include <linux/moduleparam.h>
#include <linux/delay.h>
#include <uapi/vha.h>
#include "vha_common.h"
#include "vha_plat.h"
#include <vha_regs.h>
static void mmu_flush(const struct device *dev,
struct vha_dev *vha, int ctx_id)
{
uint64_t inval = VHA_CR_OS(MMU_CTRL_INVAL_PC_EN) |
VHA_CR_OS(MMU_CTRL_INVAL_PD_EN) |
VHA_CR_OS(MMU_CTRL_INVAL_PT_EN);
/* No need to handle mmu cache, when core is already offline */
if (vha->state == VHA_STATE_OFF)
return;
#if defined(HW_AX3)
{
uint64_t pend = VHA_CR_OS(MMU_CTRL_INVAL_STATUS_PENDING_EN);
#ifdef VHA_SCF
if (vha->hw_props.supported.parity &&
!vha->parity_disable) {
/* If pending bit is set then parity bit must be set as well ! */
pend |= VHA_CR_OS(MMU_CTRL_INVAL_STATUS_PARITY_EN);
}
#endif
IOPOLL64_PDUMP(0, 20, 150, pend, VHA_CR_OS(MMU_CTRL_INVAL_STATUS));
}
#endif
if (unlikely(ctx_id == VHA_INVALID_ID))
inval |= VHA_CR_OS(MMU_CTRL_INVAL_ALL_CONTEXTS_EN);
else {
inval |= ctx_id << VHA_CR_OS(MMU_CTRL_INVAL_CONTEXT_SHIFT);
}
dev_dbg(dev, "%s: ctx_id:%d (0x%llx)\n", __func__, ctx_id, inval);
img_pdump_printf("-- MMU invalidate TLB caches\n");
IOWRITE64_PDUMP(inval, VHA_CR_OS(MMU_CTRL_INVAL));
}
/* this function is called from img_mmu, to handle cache issues */
int vha_mmu_callback(enum img_mmu_callback_type callback_type,
int buf_id, void *data)
{
struct vha_session *session = data;
struct vha_dev *vha = session->vha;
int ctx_id;
int ret = 0;
if (!vha)
return 0;
for (ctx_id = 0; ctx_id < ARRAY_SIZE(session->mmu_ctxs); ctx_id++)
mmu_flush(vha->dev, vha, session->mmu_ctxs[ctx_id].hw_id);
#if defined(VHA_MMU_MIRRORED_CTX_SUPPORT) && defined(HW_AX3)
{
/* Need to flush auxilary hw context */
int hw_id = session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].hw_id +
VHA_MMU_AUX_HW_CTX_SHIFT;
mmu_flush(vha->dev, vha, hw_id);
}
#endif
return ret;
}
static void do_mmu_ctx_setup(struct vha_dev *vha,
uint8_t hw_id, int pc_bufid, uint32_t pc_baddr)
{
img_pdump_printf("-- Setup MMU context:%d\n", hw_id);
IOWRITE64_PDUMP(hw_id, VHA_CR_OS(MMU_CBASE_MAPPING_CONTEXT));
if (!vha->mmu_base_pf_test) {
IOWRITE64(vha->reg_base, VHA_CR_OS(MMU_CBASE_MAPPING), pc_baddr);
/* This is physical address so we need use MEM_OS0:BLOCK tag
* when pdump'ing. */
img_pdump_printf("-- Setup MMU base address\n"
"WRW "_PMEM_":$0 "_PMEM_":BLOCK_%d:0 -- 'PC'\n"
"SHR "_PMEM_":$0 "_PMEM_":$0 %d\n"
"WRW64 :REG:%#x "_PMEM_":$0\n", pc_bufid,
IMG_MMU_PC_ADDR_SHIFT,
VHA_CR_OS(MMU_CBASE_MAPPING));
dev_dbg(vha->dev, "%s: setting hardware ctx id:%u\n", __func__, hw_id);
} else
dev_info(vha->dev, "Bringup test: force MMU base page fault\n");
}
int vha_mmu_setup(struct vha_session *session)
{
struct vha_dev *vha = session->vha;
int ctx_id;
for (ctx_id = 0; ctx_id < ARRAY_SIZE(session->mmu_ctxs); ctx_id++)
dev_dbg(vha->dev,
"%s: mode:%d session ctxid:%x active ctxid:%x\n",
__func__, vha->mmu_mode,
session->mmu_ctxs[ctx_id].id,
vha->active_mmu_ctx);
if (vha->mmu_mode == VHA_MMU_DISABLED) {
img_pdump_printf("-- MMU bypass ON\n");
IOWRITE64_PDUMP(VHA_CR_OS(MMU_CTRL_BYPASS_EN),
VHA_CR_OS(MMU_CTRL));
return 0;
}
/* Using model context to track active context */
if (session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].id == vha->active_mmu_ctx)
return 0;
img_pdump_printf("-- MMU_SETUP_BEGIN\n");
img_pdump_printf("-- MMU bypass OFF\n");
IOWRITE64_PDUMP(0, VHA_CR_OS(MMU_CTRL));
for (ctx_id = 0; ctx_id < ARRAY_SIZE(session->mmu_ctxs); ctx_id++) {
do_mmu_ctx_setup(vha, session->mmu_ctxs[ctx_id].hw_id,
session->mmu_ctxs[ctx_id].pc_bufid,
session->mmu_ctxs[ctx_id].pc_baddr);
/* If there are multiple sessions using the same mmu hardware context
* we need to flush caches for the old context (id is the same).
* This will happen when number of processes is > VHA_MMU_MAX_HW_CTXS */
if (vha->mmu_ctxs[session->mmu_ctxs[ctx_id].hw_id] > 1)
mmu_flush(vha->dev, vha, session->mmu_ctxs[ctx_id].hw_id);
}
#if defined(VHA_MMU_MIRRORED_CTX_SUPPORT) && defined(HW_AX3)
{
/* Need to program auxilary hw context to
* point the same page tables as base context */
int hw_id = session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].hw_id +
VHA_MMU_AUX_HW_CTX_SHIFT;
do_mmu_ctx_setup(vha, hw_id,
session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].pc_bufid,
session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].pc_baddr);
if (vha->mmu_ctxs[session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].hw_id] > 1)
mmu_flush(vha->dev, vha, hw_id);
}
#endif
/* Using model context to track context change */
vha->active_mmu_ctx = session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].id;
dev_dbg(vha->dev, "%s: update ctx id active:%x pc:%#x\n",
__func__, vha->active_mmu_ctx,
session->mmu_ctxs[VHA_MMU_REQ_MODEL_CTXID].pc_baddr <<
VHA_CR_OS(MMU_CBASE_MAPPING_BASE_ADDR_ALIGNSHIFT));
img_pdump_printf("-- MMU_SETUP_END\n");
return 0;
}
void vha_mmu_status(struct vha_dev *vha)
{
const char levels[][5] = {"PT", "PD", "PC", "BASE"};
uint64_t status1 = IOREAD64(vha->reg_base,
VHA_CR_OS(MMU_FAULT_STATUS1));
uint64_t status2 = IOREAD64(vha->reg_base,
VHA_CR_OS(MMU_FAULT_STATUS2));
uint64_t addr = VHA_CR_GETBITS_OS(MMU_FAULT_STATUS1, ADDRESS, status1);
uint8_t level = VHA_CR_GETBITS_OS(MMU_FAULT_STATUS1, LEVEL, status1);
uint8_t req_id = VHA_CR_GETBITS_OS(MMU_FAULT_STATUS1, REQ_ID, status1);
uint8_t ctx = VHA_CR_GETBITS_OS(MMU_FAULT_STATUS1, CONTEXT, status1);
uint8_t rnw = VHA_CR_GETBITS_OS(MMU_FAULT_STATUS1, RNW, status1);
uint8_t type = VHA_CR_GETBITS_OS(MMU_FAULT_STATUS1, TYPE, status1);
uint8_t fault = VHA_CR_GETBITS_OS(MMU_FAULT_STATUS1, FAULT, status1);
uint8_t bif_id = VHA_CR_GETBITS_OS(MMU_FAULT_STATUS2, BIF_ID, status2);
uint8_t tlb_entry = VHA_CR_GETBITS_OS(MMU_FAULT_STATUS2, TLB_ENTRY, status2);
uint8_t slc_bank = VHA_CR_GETBITS_OS(MMU_FAULT_STATUS2, BANK, status2);
uint64_t mapping = 0;
/* Select context and read current pc */
IOWRITE64(vha->reg_base, VHA_CR_OS(MMU_CBASE_MAPPING_CONTEXT), ctx);
mapping = IOREAD64(vha->reg_base, VHA_CR_OS(MMU_CBASE_MAPPING));
/* false alarm ? */
if (!fault)
return;
dev_dbg(vha->dev, "%s: MMU FAULT: s1:%llx s2:%llx\n",
__func__, status1, status2);
dev_warn(vha->dev, "%s: MMU fault while %s @ 0x%llx\n",
__func__, (rnw) ? "reading" : "writing", addr << 4);
dev_warn(vha->dev, "%s: level:%s Requestor:%x Context:%x Type:%s\n",
__func__, levels[level], req_id, ctx,
(type == 0) ? "VALID" :
(type == 2) ? "READ-ONLY" :
"UNKNOWN");
dev_warn(vha->dev, "%s: bif_id:%x tlb_entry:%x slc_bank:%x\n",
__func__, bif_id, tlb_entry, slc_bank);
dev_warn(vha->dev, "%s: current mapping@context%d:%#llx\n",
__func__, ctx,
mapping <<
VHA_CR_OS(MMU_CBASE_MAPPING_BASE_ADDR_ALIGNSHIFT));
}

191
drivers/nna/vha/single/vha_regs.h Normal file
View File

@@ -0,0 +1,191 @@
/*!
*****************************************************************************
* Copyright (c) Imagination Technologies Ltd.
*
* The contents of this file are subject to the MIT license as set out below.
*
* Permission is hereby granted, free of charge, to any person obtaining a
* copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction, including without limitation
* the rights to use, copy, modify, merge, publish, distribute, sublicense,
* and/or sell copies of the Software, and to permit persons to whom the
* Software is furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*
* Alternatively, the contents of this file may be used under the terms of the
* GNU General Public License Version 2 ("GPL")in which case the provisions of
* GPL are applicable instead of those above.
*
* If you wish to allow use of your version of this file only under the terms
* of GPL, and not to allow others to use your version of this file under the
* terms of the MIT license, indicate your decision by deleting the provisions
* above and replace them with the notice and other provisions required by GPL
* as set out in the file called "GPLHEADER" included in this distribution. If
* you do not delete the provisions above, a recipient may use your version of
* this file under the terms of either the MIT license or GPL.
*
* This License is also included in this distribution in the file called
* "MIT_COPYING".
*
*****************************************************************************/
#include "../vha_io.h"
#if defined(HW_AX2)
#include <hwdefs/vha_cr_mirage.h>
#elif defined(HW_AX3)
#include <hwdefs/vha_cr_aura.h>
#else
#error "No HW layout defined"
#endif
#if defined(CFG_SYS_MAGNA)
#include <hwdefs/magna_system.h>
#elif defined(CFG_SYS_VAGUS)
#include <hwdefs/vagus_system.h>
#elif defined(CFG_SYS_AURA)
#include <hwdefs/aura_system.h>
#elif defined(CFG_SYS_MIRAGE)
#include <hwdefs/mirage_system.h>
#endif
/* HW Series AURA or MIRAGE */
#if defined(HW_AX2)
#define HW_SERIES (23U)
#elif defined(HW_AX3)
#if defined(CONFIG_VHA_NEXEF)
/* 3NX-F use a different B. value */
#define HW_SERIES (32U)
#else
#define HW_SERIES (28U)
#endif
#else
#error "No HW Series defined"
#endif
/* Events macros definition */
#define VHA_EVENT_TYPE(name) \
VHA_CR_VHA_EVENT_TYPE_VHA_##name##_EN
#if defined(HW_AX2)
#define VHA_CNN_ERR_EVNTS (VHA_EVENT_TYPE(CNN0_ERROR) |\
VHA_EVENT_TYPE(CNN0_MEM_WDT) |\
VHA_EVENT_TYPE(CNN0_WDT))
#define VHA_CORE_EVNTS (VHA_EVENT_TYPE(MMU_PAGE_FAULT) |\
VHA_EVENT_TYPE(AXI_ERROR))
#elif defined(HW_AX3)
#define VHA_CNN_ERR_EVNTS (VHA_EVENT_TYPE(CNN0_ERROR) |\
VHA_EVENT_TYPE(CNN0_MEM_WDT))
#ifdef VHA_SCF
#define VHA_CORE_EVNTS ( \
VHA_EVENT_TYPE(MMU_PARITY_ERROR) |\
VHA_EVENT_TYPE(PARITY_ERROR) |\
VHA_EVENT_TYPE(LOCKSTEP_ERROR) |\
VHA_EVENT_TYPE(READY) |\
VHA_EVENT_TYPE(ERROR) |\
VHA_EVENT_TYPE(HL_WDT) |\
VHA_EVENT_TYPE(MMU_PAGE_FAULT) |\
VHA_EVENT_TYPE(AXI_ERROR))
#else /*!VHA_SCF */
#define VHA_CORE_EVNTS ( \
VHA_EVENT_TYPE(READY) |\
VHA_EVENT_TYPE(ERROR) |\
VHA_EVENT_TYPE(HL_WDT) |\
VHA_EVENT_TYPE(MMU_PAGE_FAULT) |\
VHA_EVENT_TYPE(AXI_ERROR))
#endif /* VHA_SCF */
#endif /* HW_AX3 */
/* ignore bottom 4 bits of CONFIG_ID: they identify different build variants */
#define VHA_CR_CORE_ID_BVNC_CLRMSK (0xfffffffffffffff0ULL)
#define VHA_CNN_CMPLT_EVNT (VHA_EVENT_TYPE(CNN0_COMPLETE))
#define VHA_CNN_EVNTS (VHA_CNN_ERR_EVNTS | VHA_CNN_CMPLT_EVNT)
#define VHA_EVNTS_DEFAULT ( ( \
VHA_CNN_EVNTS | VHA_CORE_EVNTS \
) & VHA_CR_OS(VHA_EVENT_ENABLE_MASKFULL))
#define VHA_SYS_CLOCK_MODE(name, mode) \
VHA_CR_SYS_CLK_CTRL0_##name##_##mode \
#define VHA_SYS_CLOCKS_DEFAULT(mode) ( (\
VHA_SYS_CLOCK_MODE(SLC, mode) \
) & VHA_CR_SYS_CLK_CTRL0_MASKFULL)
/* Clocks macros definition */
#define VHA_MAIN_CLOCK_MODE(name, mode) \
VHA_CR_CLK_CTRL0_##name##_##mode \
#if defined(HW_AX2)
#define VHA_MAIN_CLOCKS_DEFAULT(mode) ( (\
VHA_MAIN_CLOCK_MODE(CNN_EWO, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_PACK, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_OIN, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_POOL, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_SB, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_XBAR, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_NORM, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_ACT, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_ACCUM, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_CNV, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_CBUF, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_IBUF, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_CMD, mode) | \
VHA_MAIN_CLOCK_MODE(CNN, mode) | \
VHA_MAIN_CLOCK_MODE(SLC, mode) | \
VHA_MAIN_CLOCK_MODE(BIF, mode) \
) & VHA_CR_CLK_CTRL0_MASKFULL)
#elif defined(HW_AX3)
#define VHA_MAIN_CLOCKS_DEFAULT(mode) ( (\
VHA_MAIN_CLOCK_MODE(CNN_CORE_XBAR, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_MMM, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_EWO, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_PACK, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_OIN, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_POOL, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_SB, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_NORM, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_ACT, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_ACCUM, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_CNV, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_CBUF, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_IBUF, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_CMD, mode) | \
VHA_MAIN_CLOCK_MODE(CNN, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_TRS_A, mode) | \
VHA_MAIN_CLOCK_MODE(CNN_TRS_B, mode) | \
VHA_MAIN_CLOCK_MODE(SLC, mode) | \
VHA_MAIN_CLOCK_MODE(BIF, mode) \
) & VHA_CR_CLK_CTRL0_MASKFULL)
#endif
/* Reset macros definition */
#define VHA_RESET_EN(name) \
VHA_CR_RESET_CTRL_VHA_##name##_EN
#define VHA_RESET_DEFAULT ( ( \
VHA_RESET_EN(SYS_SOFT_RESET) | \
VHA_RESET_EN(AXI_SOFT_RESET) | \
VHA_RESET_EN(CNN0_SOFT_RESET) | \
VHA_RESET_EN(SLC_SOFT_RESET) | \
VHA_RESET_EN(BIF_SOFT_RESET) | \
VHA_RESET_EN(SOFT_RESET) \
) & VHA_CR_RESET_CTRL_MASKFULL)
/* NN_SYS register macros */
#define NN_SYS_CR_BASE \
(_REG_NNSYS_START)
#define NN_SYS_CR(reg) \
(_REG_NNSYS_START + NN_SYS_CR_##reg)

1044
drivers/nna/vha/vha_api.c Normal file
View File

File diff suppressed because it is too large Load Diff

2606
drivers/nna/vha/vha_common.c Normal file
View File

File diff suppressed because it is too large Load Diff

1121
drivers/nna/vha/vha_common.h Normal file
View File

File diff suppressed because it is too large Load Diff

1900
drivers/nna/vha/vha_dbg.c Normal file
View File

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More